Nerd @ Work

When Salesforce is life!

Author: Nikos Mitrakis

Some facts about Nikos:

- Salesforce Developer at Johnson & Johnson EMEA Development Centre (EDC)
- Started his Salesforce journey in 2014
- Has passed 9 certifications, including Certified Application Architect
- Holds a Physics degree
- Leader of Limerick, Ireland Developer Group
- Married since 1994, has a daughter
- Loves watching sci-fi movies and good comedies
- Lives in Limerick, Ireland

Who needs so many records?

By

On October 20, 2020

In Post

Today’s post has been written by Nikos Mitrakis, the creator of Forceea, an amazing Data Factory Framework for Salesforce.
Some facts about Nikos:
– Salesforce Developer at Johnson & Johnson EMEA Development Centre (EDC)
– Started his Salesforce journey in 2014
– Has passed 13 certifications, including Application & System Architect
– Holds a Physics degree
– Married since 1994, has a daughter
– Loves watching sci-fi movies and good comedies
– Lives in Limerick, Ireland

A first question you probably have when you read about creating millions of records is “Who really needs to create millions of records?” Sometimes it’s not “millions”; it’s anything between a few thousands to hundreds of thousands of records. But the need is the same: a flexible tool that can insert (and delete of course) many SObject records and will allow:

  • Companies of any size create sandboxes for User Acceptance Testing (UAT).
  • AppExchange ISV/Consulting partners create orgs with sample data for demos or for a realistic simulation of their app.
  • Testers or business users generate their testing data in a sandbox.
  • Architects create Large Data Volumes (LDV) for stress testing of their designs.

Forceea overview

Forceea data factory (a GitHub project) can create data using the Dadela data generation language. The framework can insert/update records synchronously for test methods (or for inserting a few hundreds of records) in your org, but it can also insert/delete records asynchronously.

Forceea has a rich set of powerful data generation tools and it’s the most sophisticated data factory for Salesforce. The latest release adds variables, permutations of serial values and the first function-x definition.

I can hear you asking: “How complex (or “difficult”) is to create records with Forceea asynchronously? Should I know to write code?“

The answer is “Yes, you should write a few lines of Apex code. But, NO, it’s not difficult at all!”. Sometimes the data creation is complex because we must have a deep knowledge of how our SObjects are related to each other, but this doesn’t need advanced programming skills.

So, what is needed to start working with it?

  • A Template.
  • An anonymous window to execute Apex scripts.
  • A Lightning component to monitor the progress.

Let’s start with..

The Template

In my previous article How to create an Apex reusable Data Factory Library using Forceea Templates, we had constructed some Templates using an older version of Forceea. The good news is that Forceea now inherently supports Templates, so the Template creation process is simpler.

What is a Template

A Template will not create data; it’s a “description” of the structure of the data we want to create.

When we construct a Template we define:

  • The SObjects that will be created.
  • The number of records of each SObject.
  • What fields will be populated.
  • The structure of field values.

A Template is a Map<String, FObject>, so our Template will start with the initialization of this Map:

Map<String, FObject> template = new Map<String, FObject>();

Defining what data we need

Before starting our Template we should have a good understanding of the SObjects and fields we need, what are the relationships between the SObjects and what data we want for each field.

Here are our (hypothetical) requirements:

Accounts

  • Record type: the record type with name MajorAccount.
  • Name: Account-1, Account-2, etc.
  • Industry: any picklist value except Banking and Services.
  • AnnualRevenue: a random integer number between 1M and 10M.
  • Rating: any picklist value.
  • Type: any random value between Prospect, Customer and Analyst.
  • Shipping address: any (real) address from U.S.

Opportunities

  • Record type: the record type with name BigOpp.
  • Name: <Account> – <text>, where <Account> is the name of the related account and <text> is a text of random words between 20 and 40 chars.
  • Amount: a random number between 10K and 1M, rounded to nearest 100.
  • StageName: any picklist value except Closed Won and Closed Lost.
  • Type: New Business.
  • CloseDate: any date between 1 Jan. 2020 and 30 June 2020.
  • AccountId: the 1st account to the 1st opportunity, the 2nd account to the 2nd opportunity and so on. If we have no more accounts, start from the 1st account, then to the 2nd, etc.

For every 1 account we’re going to create 10 opportunities.

The template for accounts

First, we “add” the Account definitions in our template:

template.put('Accounts', new FObject(Account.SObjectType)
  .setNumberOfRecords(10)
  .setDefinition(Account.Name, 'static value(Account-)')
  .setDefinition(Account.Name, 'serial type(number) from(1) step(1) scale(0)')
  .setDefinition(Account.Industry, 'random type(picklist) except(Banking,Services)')
  .setDefinition(Account.AnnualRevenue, 'random type(number) from(1000000) to(10000000) scale(0)')
  .setDefinition(Account.Rating, 'random type(picklist)')
  .setDefinition(Account.Type, 'random type(list) value(Prospect,Customer,Analyst)')
  .setDefinition(Account.ShippingStreet, 'random type(street) group(shipping)')
  .setDefinition(Account.ShippingPostalCode, 'random type(postalCode) group(shipping)')
  .setDefinition(Account.ShippingCity, 'random type(city) group(shipping)')
  .setDefinition(Account.ShippingState, 'random type(state) group(shipping)')
  .setDefinition(Account.ShippingCountry, 'random type(country) group(shipping)')
);
  • The order of the field definitions is important! Forceea generates the values for the first field definition, then for the second, etc.
  • The Name field has 2 definitions. The first generates the same (static) value “Account-” and the second serial numbers (1,2,3,..)
  • We “grouped” all address definitions in order to “link” the correct street to the correct city, postal cod, etc.
  • If we had a Billing address, we could copy the value from the Shipping, e.g. setDefinition(Account.BillingCity, 'copy field(ShippingCity)')

The Template for opportunities

Now we are going to set the Opportunity definitions:

template.put('Opportunitites', new FObject(Opportunity.SObjectType)
  .setNumberOfRecords(100)
  .setDefinition(Opportunity.AccountId, 'serial lookup(Account) mode(cyclical) source(forceea)')
  .setDefinition(Opportunity.Name, 'copy field(AccountId) from(Account.Name)')
  .setDefinition(Opportunity.Name, 'static value(" - ")')
  .setDefinition(Opportunity.Name, 'random type(text) minLength(20) maxLength(40)')
  .setDefinition(Opportunity.Amount, 'random type(number) from(10000) to(1000000) scale(2)')
  .setDefinition(Opportunity.StageName, 'random type(picklist) except(Closed Won,Closed Lost)')
  .setDefinition(Opportunity.Type, 'static value(New Business)')
  .setDefinition(Opportunity.CloseDate, 'random type(date) from(2020-01-01) to(2020-6-30)')
);

The FObjectAsync class

Now we can proceed with the actual insertion of records. Our main tool is the FObjectAsync class.

How the async process works

When we insert or delete records asynchronously, Forceea uses Queueable Apex to execute one or more jobs. These jobs have some higher governor limits (e.g. 60,000ms total CPU time and 200 SOQL queries), which is definitely positive for our data generation needs.

If you think “I’m going to create x accounts and y opportunities”, forget this way. Forceea works with iterations! An iteration is the number of records (for each SObject) defined in the Template we use. Our template creates 10 accounts and 100 opportunities, so 1 iteration will create 10 accounts and 100 opportunities.

Another important detail is Partitioning, which has two parts:

  • Template: you define the Partition field for each SObject with the method setPartitionFieldName.
  • FObjectAsync: you define the Partition field value for all SObjects with the method setPartitionFieldValue.

The Partition field value should be a string which will identify (or “partition”) the inserted records. As a best practice, use a value with a few characters, even a single letter (uppercase or lowercase).

When inserting records, Forceea checks:

  • If there is a Partition field defined in each SObject.
  • If there is a Partition field value.

If both conditions are valid, Forceea will insert the value in the partition field of each record. So, let’s say that the Partition field for Account is ForceeaPartition__c and the Partition field value is df. In this case, Forceea will insert the value:
• df1 into the records inserted in Job 1.
• df2 into the records inserted in Job 2.
• df3 into the records inserted in Job 3.
etc.

Insert records asynchronously

Now we are going to insert 1,000 iterations, so we’ll insert 1,000 x 10 = 10K accounts and 1,000 x 100 = 100K opportunities.

Open an Anonymous Apex window and enter the following lines:

new FObjectAsync(template)
    .setNumberOfIterations(1000)
    .setNumberOfJobs(20)
    .setPartitionFieldValue('df')
    .insertRecords();
  • The default number of (parallel asynchronous) jobs is 30. Here we require 20 jobs.
  • The partition value is “df”.

Execute the code and then go to the Data Factory tab of the Forceea Lightning app.

  • In the Log panel Forceea displays information about the execution of each job.
  • The Messages panel contains an overview of the async process.
  • The Progress panel will let you know how many iteration have been inserted.
  • Finally, the Job Status panel displays a visual indication of the status for each job (black: pending, green: successful, red: failure, orange: terminated).

Forceea will follow this procedure during the async insertion process:

  • Benchmarks the operation by inserting 1 iteration in the first batch. The transaction is rolled back, so it doesn’t permanently insert any records.
  • Executes the second batch of any job, which creates and insert records of each SObject defined in the Template, with as many iterations as possible (remember the benchmarking).
  • If there are no errors and there are more iterations to be inserted, a third batch is created, and so on.
  • When all iterations assigned to a job have been inserted, the job ends with a successful completion.

When we have a serial definition, Forceea will insert the records without any gaps in the serialization!

Delete records asynchronously

The deletion process follows almost the same logic:

new FObjectAsync(template)
    .setNumberOfJobs(20)
    .setPartitionFieldValue('df')
    .deleteRecords();

Execute the above Apex code and then go to the Data Factory tab to watch the progress.

Forceea will follow these steps during the async deletion process:

  • Reverses the order of SObjects in the Template, so the last SObject will get the first position, etc.
  • If all SObjects in the Template have a Partition field and FObjectAsync has a Partition field value, a number of jobs are enqueued for parallel processing (each job will delete all records of different partitions), otherwise it enqueues only 1 job (no partitioning).
  • The deletion starts from the SObject in the first position, executing the first batch of each job, which benchmarks the transaction to calculate the maximum number of records that can be deleted in every batch. This first benchmarking batch deletes up to 200 records.
  • If there are no errors and there are more records to be deleted, a second batch is created after the completion of the first batch, and so on.
  • When all SObject records assigned to a job have been deleted, the job moves to the second SObject, etc.

Important: if Forceea finds in the Template a definition for the RecordTypeId field of an SObject, it will delete the records of this Record Type only.

Forceea will stop the execution of a job when an error is encountered, except from the errors related to record locking, where it will raise an error only after the 5th occurrence of the UNABLE_TO_LOCK_ROW error.

Using existing lookup records

Forceea will take care of all the complex orchestration of the asynchronous process. The parallel processing offers an advantage, but it’s based on the assumption that we won’t query any existing records from the database, otherwise we may have record locking.

For example, if we have a custom SObject Language__c and we have the lookup field Language__c on Opportunity, to get random IDs for this field we would use:

setDefinition(Opportunity.Language__c, 'random lookup(Language__c) source(salesforce)')

If the above definition raises the UNABLE_TO_LOCK_ROW error (unable to obtain exclusive access to this record), then your only option is to use 1 job only with setNumberOfJobs(1).

Conclusion

Nobody can say that data generation is simple or without issues. Under the hood, the data generation process is quite complex, but it shouldn’t be to the user; Forceea will gracefully handle all the complexity.

I strongly believe that an admin, a tester or even a business user, with no Apex knowledge, can insert/delete records asynchronously using FObjectAsync and existing Templates, which a developer or advanced admin could create.

You can find the code of the above scripts in Forceea-training GitHub repo. And don’t forget to read the Forceea Success Guide; it has a lot of examples and details.

How to create an Apex reusable Data Factory Library using Forceea Templates

By

On August 21, 2019

In Post

Today’s post has been written by Nikos Mitrakis, the creator of Forceea, an amazing Data Factory Framework for Salesforce.

Some facts about Nikos:

– Salesforce Developer at Johnson & Johnson EMEA Development Centre (EDC)
– Started his Salesforce journey in 2014
– Has passed 10 certifications, including Certified Application Architect
– Holds a Physics degree
– Leader of Limerick, Ireland Developer Group
– Married since 1994, has a daughter
– Loves watching sci-fi movies and good comedies
– Lives in Limerick, Ireland

Introduction to a familiar situation

Let’s face it with honesty and courage: writing test methods for our Apex code is (very often) boring and time-consuming, usually because of the difficulty regarding the creation of test data. In most implementations we can see a mediocre result, which could be described as a class (e.g. DataFactory) including static methods that generate and insert SObject records. The following code describes this inefficient pattern:

public class DataFactory {
    public static List<Account> createAccounts(Integer numRecords) {
        List<Account> results = new List<Account>();
        for (Integer counter = 1; counter <= numRecords; counter++) {
            Account record = new Account();
            // define fields
            record.Name = ...
            ...
            results.add(record);
        }
        return results;
    }
    // other methods
}

Using this logic, we can execute our Data Factory method with:

List<Account> accounts = DataFactory.createAccounts(100);
insert accounts;

to create and insert 100 accounts. As you can see, one issue here is how we generate data for other fields in our test method. But things get worse when we create related SObjects, for example contacts with accounts. Let’s examine a new method createContacts, based on the previous pattern:

public static List<Contact> createContacts(Integer numRecords) {
    List<Account> accounts = createAccounts(10);
    // optionally process accounts and manually add/modify Account fields
    insert accounts;
    List<Contact> results = new List<Contact>();
    for (Integer counter = 1; counter <= numRecords; counter++) {
        Contact record = new Contact();
        // define fields
        record.LastName = ...
        record.AccountId = ... // get the ID from accounts list
        ...
        results.add(record);
    }
    return results;
}

When we call the above method from our test method, e.g. with

List<Contact> contacts = DataFactory.createContacts(100);
//  optionally process contacts and manually add/modify Contact fields
insert contacts;

we certainly insert 10 accounts and 100 contacts related to these accounts. But what if we need to modify the generated accounts or we need to insert additional Account fields? This pattern doesn’t allow to do this. In more complex scenarios, we may have to insert many more SObjects. The final result is a Data Factory class with methods that create test data BUT without the ability to easily modify the created records.

I can finally hear your question: Do you propose a better approach? Is there a more flexible and easier way to do it? And the answer is YES!

Making a Data Factory Library with Forceea Templates

Forceea Data Factory framework is an open source GitHub project, with the following capabilities:

  • creates records for standard or custom objects, for any standard or custom field
  • automatically definines the required fields
  • creates static or random data for fields of any data type: Integer, Currency, Double, Date, Datetime, Time, Boolean, String, TextArea, Percent, Reference, Email, Phone, URL, Base64 (BLOB), Picklist and MultiPicklist
  • creates real random first and last names
  • creates real random addresses with street, zip code, city, region/state and country
  • creates serial data for date, datetime, integer, decimal, currency and percent
  • can copy data from another field of the same record or a lookup record
  • can create the same random data, using a pseudo-random number generator
  • handles record types and field dependencies (dependent picklists)
  • supports record groups for inserting and deleting records
  • validates the definitions based on the field data type
  • provides many methods to get/insert the created records, add/delete field definitions, get the errors, configure the amount of information returned during run-time (debug log) and more
  • includes an extended error messaging system

and will be our main tool to build a powerful and flexible DataFactory class (our Data Factory Library). This class will include static methods, our Templates, which actually will not insert any data at all! What these Templates do is to instruct Forceea how to generate the data.

Let’s meet our first Template:

public class DataFactory {
    // returns definitions for: Accounts 
    public static FObject getDefAccounts() {
        FObject result = new FObject('Account');
        result.setDefinition('Name', 'static value(Company)');
        result.setDefinition('Name', 'serial type(number) from(1) step(1) scale(0)');
        result.setDefinition('Phone', 'random type(phone) format("(30) DDD dD-00-DD")');
        result.setDefinition('Industry', 'random type(picklist)');
        result.setDefinition('Site', 'random type(url)');
        return result;
    }
}

Obviously the method getDefAccounts returns an FObject – the class instance for generating data with Forceea. Reading the code you can see that we define accounts with random values for all required fields. So, these are our guidelines so far:

  • Create a DataFactory class
  • Create a master Template for each SObject, with the name getDef<SObjectApiName>s, e.g. getDefCases
  • Use the above pattern for each master Template, defining all common required fields (the fields required by any record type)
  • For the Template field definitions, use definitions which generate
    – random values for picklist fields
    – random values for fields with date/datetime, checkbox, email, phone, currency, percent, address, and text area data types
    – serial values for the Name field (notice how we did it in the getDefAccounts method)

Even though it’s not obvious from the above code, Forceea (by default) will find and insert field definitions for any required fields we haven’t defined, but it’s a best practice to define all these required fields in our master Template.

The setDefinition method sets the expected values for each field, using a descriptive data generation language called Dadela. For example, the definition random type(picklist) except(Hot) for the Rating field generates random values from the pisklist field’s values, excluding the value “Hot”.

Now, for every record type of each SObject create a new Template, for example:

// returns definitions for: Accounts with MediumAccount record type
public static FObject getDefMediumAccounts() {
    FObject result = getDefAccounts();
    result.setDefinition('RecordTypeId', 'static value(MediumAccount)');
    result.setDefinition('NumberOfEmployees', 'random type(number) from(10) to(100) scale(-1)');
    result.setDefinition('AnnualRevenue', 'random type(number) from(1000000) to(10000000) scale(3)');
    return result;
}

This new Template builds on the master Template getDefAccounts, defining only the record type and the additional fields which are related to this specific record type (NumberOfEmployees and AnnualRevenue). All other defined fields from the master Template are used as they are, so we don’t duplicate any field definitions. Our additional guideline:

  • Create a record type Template for each SObject’s record type, with the name getDef<RecordTypeDescription><SObjectApiName>s, e.g. getDefServiceCases

This is it! This is what we need to do for SObjects which don’t include a Lookup or Master-detail relationship. But how do we create templates for those SObjects that they do include a relationship? Let’s see how, with our second SObject:

// returns definitions for: Accounts - Contacts
public static Map<String, FObject> getDefContactsAndAccounts() {
    // initialize
    Map<String, FObject> fobjectsByName = new Map<String, FObject>();
    String objName = '';
    // Account
    objName = 'Account';
    FObject objAccount = getDefMediumAccounts(); 
    fobjectsByName.put(objName, objAccount);
    // Contact
    objName = 'Contact';
    FObject objContact = new FObject(objName);
    objContact.setDefinition('FirstName', 'random type(firstname) group(name)');
    objContact.setDefinition('LastName', 'random type(lastname) group(name)');
    objContact.setDefinition('AccountId', 'random lookup(Account) source(forceea)');
    objContact.setDefinition('LeadSource', 'random type(picklist)');
    objContact.setDefinition('Title', 'random type(list) value(Developer, CFO, Account Manager, CEO, Logistics Manager)');
    objContact.setDefinition('Email', 'random type(email)');
    fobjectsByName.put(objName, objContact);

    return fobjectsByName;
 }

I think you’ll agree that this Template is more interesting. Its first lines use the previously created record type Template getDefMediumAccounts to define the Account fields. We could also

  • insert one or more new field definitions using objAccount.setDefinition('FieldApiName', '<Field Definition>') before fobjectsByName.put(..) or
  • modify an existing field definition – for example to define a static value for the (existing) NumberOfEmployees field, we can use
    // delete previous field definitions
    objAccount.deleteFieldDefinitions('NumberOfEmployees');
    // add new definition
    objAccount.setDefinition('NumberOfEmployees', 'static value(100)');

Finally, we insert the FObject for the Account into the fobjectsByName map and we proceed to the field definitions for contacts.

If you noticed the definition objContact.setDefinition('AccountId', 'random lookup(Account) source(forceea)' and asked yourself what is source(forceea), this is a way to request that the framework will get the related account IDs from the previously inserted accounts. There are a lot of lookup field definitions that will certainly need your attention if you start developing with the framework, but for the moment let’s not go any deeper.

In many implementations we have a set of dependent SObjects. Let’s say that in order to create records for the Case, we have to create records for Account and Contact (and perhaps records for 4 more other SObjects) using a Template like getDefCasesAndAccountsContacts. This is a kind of quite complex data factory process, which can be handled by Forceea very smoothly – you just add the following pattern for each requird SObject:

objName = '<SObjectApiName>';
FObject objMySObject = new FObject(objName);
objMySObject.setDefinition('<FieldApiName>', '<Field Definition>');
// other required field definitions
fobjectsByName.put(objName, objMySObject);

Our last guidelines:

  • Document the SObjects that are returned by any Template, with the correct order, e.g. // returns definitions for: Accounts - Contacts - Cases
  • Use the format getDef<SObjectName>And<RelatedSObjects> for Templates with related SObjects, e.g. getDefCasesAndAccountsContacts

Finally, insert the following method in your DataFactory class:

public static void insertRecords(Map<String, Fobject> fobjects) {
    for (FObject obj: fobjects.values()) {
      obj.insertRecords(true);
    }
}

The test method

After you have created your Templates, let’s see how you can take full advantage of them. We’ll use getDefContactsAndAccounts as an example. In your test method, the first step is to define a map:

Map<String, FObject> fObjects = DataFactory.getContactsAndAccounts();

The second step is to modify any SObject definitions, if it’s needed. For our example here, we’ll make things a little more difficult with the following requirements:

  • For the Account: we need to insert 10 records, with a) random values for the Description field and b) any picklist value except “Hot” for the Rating field.
  • For the Contact: we need to insert 100 records, with the random values from 1/1/1960 to 31/12/2000 for the Birthdate field.
  • All other fields will get the default definitions from the their Templates.
// initialize
Map<String, FObject> fobjectsByName = DataFactory.getDefContactsAndAccounts();
FObject objAccount = fobjectsByName.get('Account');
FObject objContact = fobjectsByName.get('Contact');
// define number of records
objAccount.records = 10;
objContact.records = 100;
// optionally modify an existing definition
objAccount.deleteFieldDefinitions('Rating');
//  optionally define new field definitions
objAccount.setDefinition('Description', 'random type(text) minlength(10) maxlength(40)');
objAccount.setDefinition('Rating', 'random type(picklist) except(Hot)');
objContact.setDefinition('Birthdate', 'random type(date) from(1960-1-1) to(2000-12-31)');
// insert records
DataFactory.insertRecords(fobjectsByName);

Using the above pattern, it’s easy for everyone to understand what changes have been made in comparison to the getDefContactsAndAccounts Template.

Did you say that we need the inserted contacts for our System.assert? No problem at all! Just use:

List<Contact> contacts = objContact.getInsertedRecords();
OR
List<Contact> contacts = FObject.getInsertedRecords('Contact');

Conclusion

Forceea Templates are easy to implement and they are powerful enough to help you write your Apex test methods faster. The most important is that

  • your test methods will be more understandable by any other developer, and
  • the new test methods will require less effort to develop

The best way to see if this solution is suitable for you is to start working with it and create a pilot version of your new DataFactory class. If you’re not satisfied with your existing Data Factory (or if you don’t have a Data Factory at all), why don’t you give it a try?

Introducing Dadela, a brand new business-oriented programming language

By

On April 18, 2019

In Post

 
Today’s post has been written by Nikos Mitrakis, the creator of Forceea, an amazing Data Factory Framework for Salesforce.

Some facts about Nikos:

  • Salesforce Developer at Johnson & Johnson EMEA Development Centre (EDC)
  • Started his Salesforce journey in 2014
  • Has passed 10 certifications, including Certified Application Architect
  • Holds a Physics degree
  • Leader of Limerick, Ireland Developer Group
  • Married since 1994, has a daughter
  • Loves watching sci-fi movies and good comedies
  • Lives in Limerick, Ireland

Introduction

Dadela (pronounced dadèla) is a new programming language, designed with the intention to improve the way we generate data.

Dedala is business-oriented (not object-oriented) and it can be used by a user without programming skills (e.g. a tester or a business analyst).

The language specification is a project initiative on GitHub.

It was originally designed for Forceea data factory framework.

In Dadela we use the following language elements: 

  • The Environment is the system (programming language or application) that hosts Dadela’s components and capabilities, sets the execution limitations and defines the integration details for sources and destinations
  • A component is a language element. There are various components: attributes, variables, lists, entity definitions, templates and operations
  • A repository is a physical (e.g. a file) or a virtual (e.g. a database record) storage for the components
  • A library is a special repository, containing templates and lists
  • A template is a blueprint for creating records of an entity
  • An entity is a database table or a matrix. It can be a Salesforce SObject, an Oracle table, an Excel worksheet, etc
  • An operation is a process that creates/inserts records or exports records in text format (JSON) or CSV)
  • A source is a one-way integration between an external database and the Environment for retrieving records
  • A destination has the opposite direction (from the Environment to an external entity for inserting/exporting records

Lists play a key role in Dadela. A list is something like (item1,  item2,  item3), using a comma to separate the list items. In fact, all repository components can be considered as list items. Some examples:

number x: 100, # a numerical variable
list myList: (100, 200, 300), # a list
template BigAccounts: ( ... ), # a template
entity Account: ( ... ), # an entity definition

Lists may contain other lists or variables. Variables are referenced using @ and lists with !

number myVar1: 100,
number myVar2: 1 + myVar1, # myVar2 = 101
list myList1: (one, two, three),
string five: five, # @five = "five"
list myList2: (!myList1, four, @five), # myList2 = {one, two, three, four, five}

Of course, comments are inserted with #.

We don’t have the space to go deeper, but you’ve got the idea: the main usage of variables is to assign the same value to different components, facilitating the construction of entity definitions.

Create a template

The best way to understand a new language is “by example”. So, let’s suppose I am a business user trying to create and insert records for Opportunities, in order to prepare data for User Acceptance Testing (UAT). In my Environment I’ll create a new repository (a library), with the name “SalesTemplates” and a template in the library for the “Opportunity” entity.

Repository: SalesTemplates

template BigOpportunities: (
    Name: copy field(AccountId) from(Description),
    Name: static value(" - "),
    Name: serial type(number) from(1) step(1) scale(0) format(000),
    Amount: random type(number) from(1M) to(10M) scale(-3),
    StageName: random type(picklist) except(Closed Won, Closed Lost)
)

 Let’s “translate” the above, line by line:  

template BigOpportunities: (
  • start the definition of a template called “BigOpportunities”  and execute the following steps for each generated record
Name: copy field(AccountId) from(Name),
  • for the values of the Opportunity field “Name” find the Account record which is related to the lookup Opportunity field “AccountId” (the AccountId field stores the ID of a related Account record)get the value of the (Account) field “Description” from this record  
Name: static value(" - "), 
  • for the values of the Opportunity field “Name”add the string ” – ” after the previous value
 Name: serial type(number) from(1) step(1) scale(0) format(000), 
  • for the values of the Opportunity field “Name”get serial numbers starting from 1, adding 1, with rounding to 0 decimal pointsformat the result like 001, 002, 003, …add the result after the previous value
 Amount: random type(number) from(1M) to(10M) scale(-3),
  • for the values of the Opportunity field “Amount” get random numbers, with minimum number 1,000,000, with maximum number 10,000,000, with rounding to 1,000 
StageName: random type(picklist) except(Closed Won, losed Lost),
  • for the values of the Opportunity field “StageName” get any value from the picklist values of this field, except from “Closed Won” and “Closed Lost”
)
  • ends the field definitions of this template

Field definitions

Dadela has many field definitions, in 4 types (commands): 

copy

  • These definitions copy the value of another field or the value of a field from a related (lookup) record

static

  • Static definitions just display a string, number, date/datetime or boolean value

serial

The serial definitions create serial values for

  • numbers, dates, datetimes and lists

random 

The random definitions create random values for

  • numbers, booleans and strings, dates and datetimes, lists and picklists, lookup fields, emails, phone numbers, URLs, addresses (street, postal code, city, province, country), first and last names, and text sentences

Create the entity definition

Now, let’s continue our example. In our Environment, we create a new repository, with the name “MyEntities”.

Repository: MyEntities

language: French, 
locality: France (North Europe, Europe, EU),  

entity Account: (
    records: 100,
    alias: BigAccounts,

    Rating: random type(picklist) except(Hot),
    Phone: random type(phone) format("(30) 210 dD-00-DD"),
    Industry: random type(picklist),
    Type: random type(list) value(Prospect, Customer - Direct, Customer - Channel), 
  
    # addresses
    ShippingStreet: random type(street) group(shipping),
    ShippingPostalCode: random type(postalcode) group(shipping),
    ShippingCity: random type(city) group(shipping),
    ShippingState: random type(state) group(shipping),
    ShippingCountry: random type(country) group(shipping)
),

entity Opportunity: (
    template: SalesTemplates.BigOpportunities,
    records: 200,
    alias: MyBigOpportunities,
    virtual MyField: random type(list) value(Closed Won, Closed Lost),
    StageName: copy from(MyField)
)

 Let’s “translate” again: 

language: French,
  • the default language of names (first/last) and addresses in this repository will be “French” 
locality: France (North Europe, Europe, EU),
  • the default geographic area (locality) in this repository will be “France”, which will be a member of other localities (“North Europe”, “Europe” and “EU”) 
entity Account: ( ... ),
  • set the field definitions for accounts with the default number of records to be 100 with random values for the fields Rating, Phone, Industry and Type with random “real” addresses from France, in French
entity Opportunity: ( template: SalesTemplates.BigOpportunities,
  • use the field definitions from template “BigOpportunities” in repository “SalesTemplates”
virtual MyField: random type(list) value(Closed Won, Closed Lost),
  • define a “virtual” field: a virtual field is like a variable
StageName: copy from(MyField)
  • get the value of the virtual field (we used this for demonstration purposes – in a real situation, we use more than one virtual fields to construct a field definition)

Note that the definition for the field “Amount” will be the same as the definition in the template (this is the purpose of a template, after all).Of course, we could have defined a list for the stages inside the entity definition, or it could be a list in the repository or in a library, e.g.

entity Opportunity: (
    list stages: (Closed Won, Closed Lost),
    ...
    StageName: random type(list) value(!stages)
)

Operations

The next step is to generate and store our data.

insert

If we want to insert the records into a “database”:

  • We configure a destination on our Environment. This destination could be a Salesforce org, or a SQL Server database.We use an insert operation:
insert MyBigOpportunities: (
    destination(MySalesforceOrg1) group(MyGroupA)
),

The (optional) group parameter assigns a “tag” to the inserted records. This grouping is very helpful when we use the inserted records for further processing.

create

Another solution could be the create operation:

create MyBigOpportunities: (group(MyGroupA)),

Now we just generate the opportunity records and we can insert or export them later.

export

And finally, for exporting to JSON or CSV:

export MyBigOpportunities: (
    destination(Ora1)
    type(json) group(MyGroupA)
),

No magic here! The Environment will export our opportunities in JSON format.

Conclusion

We had a very short introduction to the syntax and capabilities of Dadela. If you are interested to learn more, you can find all the details on the GitHub project.

Closing this article, I’d like to remind that the language specifications are open for discussion and contribution to the further development and application of the language.

[Salesforce DX / Data Fractory] Populate your Salesforce org with data using Forceea and Salesforce CLI: a step-by-step guide

By

On October 23, 2018

In Post

 
Today’s post has been written by Nikos Mitrakis, the creator of Forceea, an amazing Data Factory Framework for Salesforce.

Some facts about Nikos:

  • Salesforce Developer at Johnson & Johnson EMEA Development Centre (EDC)
  • Started his Salesforce journey in 2014
  • Has passed 9 certifications, including Certified Application Architect
  • Holds a Physics degree
  • Leader of Limerick, Ireland Developer Group
  • Married since 1994, has a daughter
  • Loves watching sci-fi movies and good comedies
  • Lives in Limerick, Ireland

All posts by Nikos

So, you are an awesome Administrator, Developer, Consultant or Architect and you have your Scratch org, Developer Sandbox or Developer edition org ready for development, testing, training or a demo. But your org doesn’t have any data ☹, or it doesn’t have the data you want. What can you do?

Well, you have the following options:

  1. Use an ETL tool to move data from another Org or get data from CSV files.
  2. Use Salesforce CLI (Command Line Interface) using the commands sfdx force:data:tree:export and force:data:tree:import to export/ data from another Org and import the data (in JSON format).
  3. Develop a custom solution (Apex) to insert the records you need programmatically.
  4. Use a Data Factory to insert the required data.

The first (ETL) and second (CLI) solutions don’t need development skills, they may not be very difficult to configure (using CLI should be easier), and they let you populate your org with a specific set of records – if you really want this. The problem is that data requirements change very fast and if you have to continuously create manually the records you need in one org and then copy them to your org, this process will eventually be slow and time-consuming.

The third (Apex) solution gives you the maximum flexibility, but it’s time-consuming as well, not to mention the high level of technical knowledge it requires.

Now let’s see the fourth solution, the Data Factory. Using a Data Factory, you won’t have to write complex code or have advanced knowledge of an ETL tool. You’ll be able to easily generate and insert the records you need for any SObject and easily modify the process to adopt any new data requirements. Sounds interesting?

Stay with me as we’re going to implement such a data population project using Forceea (forˈsēa) data factory framework – this framework is an open source GitHub project (https://github.com/nmitrakis/Forceea). Keep in mind that the framework is perfect for a developer who wants to create records for the test methods, but that’s another story.

In this step-by-step guide we’ll assume that you have no programming knowledge! (you may be even a power Salesforce user). Ready to start? (Take a deep breath…)

Step 0: Prerequisites – Prepare your org

  • For Classic orgs: You have created a Sandbox or a Developer org.
  • For Salesforce DX Scratch orgs: you have created your scratch org.

For both Classic and DX, after the org creation you should have deployed your project’s metadata and set your org ready for the final data population.

Step 1: Install the Salesforce CLI

If you haven’t installed the Salesforce CLI, go to https://developer.salesforce.com/tools/sfdxcli, download the CLI for your system and install it. Accept the default settings and after a while your Salesforce CLI will be ready to run.

Step 2: Authorize your org

To use CLI, you need to authorize your org. To do this, open the Command Prompt window and execute the following command:

sfdx force:auth:web:login -r https://login.salesforce.com -a OrgAlias

Some comments here:

  • If you’re authorizing a sandbox, the instance URL will be https://test.salesforce.com.
  • You should replace OrgAlias with the alias of your org (choose one, e.g. sf1).

To verify that everything is fine, log into your org using the command

sfdx force:org:open -u OrgAlias

Step 3: Install Forceea

There are two ways to install Forceea.

Method 1: Go to https://github.com/nmitrakis/Forceea and click the following button:

You’ll be re-directed to the page:

  • If you install in a sandbox, select Deploy to: Sandbox. Otherwise, accept the default selection (Production/Developer).
  • Click the Login to Salesforce button (on the right corner) and give your credentials.
  • Then click Allow to give the tool the access to your org. You are redirected to a new page where the From GitHub Repository section lists the details of Forceea URL and the To Salesforce Org section the details of the target org. Click the Deploy button (on the right corner).
  • After a few seconds you’ll see the message Deployment Complete at the bottom of your page.

Method 2: Go to the GitHub repository https://github.com/nmitrakis/Forceea and click Clone or Download.

Click Download ZIP. After a few seconds the Forceea-master.zip will have been downloaded. Unzip the zip file and you will find the metadata folder, which contains

  • the src folder, with a folder with each metadata component and the package.xml
  • the dx folder, with the Salesforce DX components
  • a build.properties file
  • a build.xml file

The easiest way to regularly use Forceea is to include its metadata into your Repository (either classic or DX metadata). Then you can deploy it with your project’s metadata.

Step 4: Create the Forceea scripts

What are Forceea scripts?

Forceea uses a language (Sample Data Definition Language – SDDL) to describe the “nature” of the required data. A script is a set of commands. Let’s see an example, which creates and inserts opportunities:

FObject obj = new FObject('Opportunity', 100);
obj.setDefinition('Name', 'static value(Opp-)');
obj.setDefinition('Name', 'serial type(number) from(1) step(1) scale(0)');
obj.setDefinition('AccountId', 'random lookup(Account) field(Industry) value(Chemicals,Construction) source(salesforce)');
obj.setDefinition('Amount', 'random type(number) from(1000000) to(10000000) scale(-3)');
obj.setDefinition('StageName', 'random type(picklist)');
obj.setDefinition('CloseDate', 'random type(date) from(2017-01-01) to(2017-12-31)');
obj.setDefinition('Forceea__c', 'static value(xJio9!23)'); // ** create this field **
obj.insertRecords(true);

If the above commands look like Apex statements, you’re right, they are Apex statements, but you may forget it for the moment! The above script will insert 100 Opportunity records with:

  • Name: Opp-1, Opp-2, .., Opp-100
  • Account: any account for which the Industry is “Chemicals” or “Construction”
  • Amount: a random integer from 1M to 10M, round to 1000 (e.g. 9,252,000)
  • Stage: any picklist value
  • Close date: any date from 1 Jan. 2017 to 31 Dec. 2017

A field definition begins with obj.setDefinition and has two text parameters:

  • The field API name, e.g. Amount or AccountId
  • The SDDL definition, e.g. random type(number) from(1000000) to(10000000) scale(-3) or random type(picklist).

Finally, we insert the created records using obj.insertRecords(true). The parameter true means that it is all or nothing, so if any error happens, no records will be inserted.

The framework has many cool features:

  • Creates records for standard or custom objects, for any standard or custom field.
  • Automatically defines required fields.
  • Can create data for fields of every data type: Integer, Currency, Double, Date, Datetime, Time, Boolean,
    String, TextArea, Percent, Reference, Email, Phone, URL, Base64, Picklist and MultiPicklist.
  • Handles record types and field dependencies (dependent picklists).
  • Supports record groups for inserting and deleting records.
  • Validates the definitions based on the field data type.
  • Can be used for in test methods or for populating Salesforce orgs with sample data for demos and User
    Acceptance Testing (UAT).
  • Has an extended error messaging system.

    • You can get a lot of script examples from http://bit.ly/Forceea131_Examples and download the User Guide from http://bit.ly/Forceea131_UserGuide.

    Write your own scripts

    Now you know what is a Forceea script, let’s see how you can write the scripts for your specific needs. To do this, use the following template:

    FObject obj = new FObject('SObject_API_Name', Number_of_records);
FObject.seed = 1;
<field definitions>
obj.insertRecords(true);

    where are one or more SDDL field definitions, like the previous definitions we used to create a script for Opportunity records.

    The seed is very important when you want to create the same random records every time. For example, when you populate a Scratch org with sample data, we may want each developer to have the same (but random) set of records. Forceea uses its own engine to create random data (generally called Pseudo-random Number generator – PRNG), which you can control.

    Now, let’s suppose you want to populate your org with Account and Opportunity records. This means that first you must create records for accounts and then for opportunities (an opportunity is related to an account, that’s why we insert accounts first).

    Accounts

    Open your favourite text editor (I use VSCode) and create a script using the above template. Your script may look like:

    FObject obj = new FObject('Account', 100);
FObject.seed = 1;
obj.setDefinition('RecordTypeId', 'static value(BigAccount)');
obj.setDefinition('Name', 'static value("Company ")');
obj.setDefinition('Name', 'serial type(number) from(100) step(1) scale(0)');
obj.setDefinition('NumberOfEmployees', 'random type(number) from(10) to(1000) scale(-1)');
obj.setDefinition('AnnualRevenue', 'random type(number) from(10000000) to(100000000) scale(3)');
obj.setDefinition('Rating', 'random type(picklist) except(Hot)');
obj.setDefinition('Phone', 'random type(phone) format("(30) 210 dD-00-DD")');
obj.setDefinition('Industry', 'random type(picklist)');
obj.setDefinition('Type', 'random type(list) value(Prospect, Customer - Direct, Customer - Channel)');
obj.setDefinition('Site', 'random type(url)');
obj.setDefinition('ShippingStreet', 'random type(street) group(shipping)');
obj.setDefinition('ShippingPostalCode', 'random type(postalcode) group(shipping)');
obj.setDefinition('ShippingCity', 'random type(city) group(shipping)');
obj.setDefinition('ShippingState', 'random type(state) group(shipping)');
obj.setDefinition('ShippingCountry', 'random type(country) group(shipping)');
obj.insertRecords(true);

    Even though we don’t have the space here to describe every SDDL definition of the above script, I’d like to mention the first field definition for field RecordTypeId (if you use record types, modify the definition with the API name of your record type instead of BigAccount, otherwise delete this line) and the address definitions ShippingStreet, ShippingPostalCode, etc (where you get real addresses from United States).

    You may think that these definitions are difficult, but most of them are obvious and easy to understand. So, back to our script for the Account SObject, give it the name Accounts.apex and save it into your preferred folder (let’s call it ForceeaScripts).

    Opportunities

    Now it’s time to create the Opportunities.apex file with the script for the Opportunity SObject. Follow the previous procedure and write a script like the example I gave you above.

    Keep in mind that Forceea will try to automatically find the required fields and generate a field definition for them but, as a rule of thumb, you should always create your definitions for all required fields.

    Record Deletion

    When you create records, you probably want to repeat this procedure again, and replace the existing set of data with another set with different or the same field values. To do this, you probably want to delete the existing inserted records before you insert the new ones. This is the responsibility of the Record Deletion files.

    First, we’ll delete the opportunities we created. Follow these steps to configure it:

    • Create a new text file, call it delete-opportunities.apex and save it in the ForceeaScripts folder.
    • Write this script in the file:

      String flag = 'xJio9!23';
List<Opportunity> records = [SELECT Id FROM Opportunity WHERE Forceea__c = :flag LIMIT 5000];
delete records;

    You’re right! At first sight these commands may look difficult if you are not a developer, but they aren’t really hard to understand if you follow these instructions:

    • Line 1: You may select your own (random) value, like xJio9!23 – but don’t forget to enclose it in single quotation marks
    • Line 2: Find a filed you don’t use, which has a Text data type or create your own text field and call it Forceea__c. Here I use the latter, because I couldn’t find any available (not used) standard field on the Opportunity SObject.

    Now go to the opportunities.apex file and add the following line before the command obj.insertRecords(true); :

    obj.setDefinition(Forceea__c', 'static value(xJio9!23)');

    As you can see, we set the field definition for the field Forceea to be our static value xJio9!23. Do the same for the Account SObject, creating the file delete-accounts.apex with the following commands:

    String flag = 'xJio9!23';
List<Account> records = [SELECT Id FROM Account WHERE Tradestyle = :flag LIMIT 5000];
delete records;

    In Line 2, I suppose the TradeStyle text field is not used in your org, so you may use it here. Else, you should create a new field as previously.

    Go to the accounts.apex file and add the following line before the command obj.insertRecords(true); :

    obj.setDefinition('TradeStyle', 'static value(xJio9!23)');

    Be careful! The above scripts will delete up to 5,000 records, so if you have more records you have 2 options:

    • Increase LIMIT 5000 up to 10000 (e.g. LIMIT 8000)
    • Execute the delete-accounts.apex again (you may use it as many times as you want)

    That’s it! Now you have 4 files into your ForceeaScript folder:

    • account.apex
    • opportunities.apex
    • delete-accounts.apex
    • delete-opportunities.apex

    For your convenience, I have created these files in a zipped file, which you can download from http://bit.ly/ForceeaWithCLI.

    You may use these files for your training. I suppose you should modify them if you want to use them in a production system (with specific requirements for the field values), but you’ll probably need no modifications if you execute them in an out-of-the-box Developer/Scratch org.

    Step 5: Execute your scripts

    The last step is to execute the scripts and insert the created records. Create a new text file, call it execute-scripts.bat and save it in the ForceeaScripts folder (I included it in the above zipped file).

    In this file you’re going to insert the CLI commands which execute the *.apex files you created previously.

    If you use Windows, your execute-scripts.bat file will look like:

    sfdx force:apex:execute -f .\delete-opportunities.apex -u OrgAlias
    sfdx force:apex:execute -f .\delete-accounts.apex -u OrgAlias
    sfdx force:apex:execute -f .\accounts.apex -u OrgAlias
    sfdx force:apex:execute -f .\opportunities.apex -u OrgAlias

    where orgAlias is the alias for your org you defined in Step 2.

    If you’re using a Mac, each command may look like:

    sfdx force:apex:execute -f /Users/Username/Documents/ForceeaScripts/accounts.apex -u OrgAlias

    When you write your Forceea scripts (e.g. your script in the accounts.apex file), you may have errors to correct. For example, you may have misspelled a field’s API name or your SDDL definition may have a syntax error (a very common source of errors is the effect of the Validation Rules and Triggers). Fortunately, the framework has an extensive debug log, with detailed error messages (see the User Guide, pages 44 – 50). You can activate these debug logs, if you insert the command obj.setVerbose(‘info’) or obj.setVerbose(‘debug’) after the first command of your accounts.apex or opportunities.apex script files.

    I suggest you initially execute your script files one by one.

    • If you already have inserted records, first execute the two delete-object files. Copy the “delete” commands from your execute-scripts.bat to your Command Prompt window and execute them. Then verify that records have been deleted successfully.
    • Copy the “insert” commands from your execute-scripts.bat file to your Command Prompt window and execute them one by one, verifying that records have been inserted successfully.
    • When you have verified that everything works as expected, you may just execute your batch file in the Command Prompt window: > execute-scripts.bat (Window users)

    If you are a Mac user, you could use a similar batch execution or just simply copy all commands of the execute-scripts file to the Command Prompt window and execute them at once.

    Because of this process, all lines of the execute-scripts file will be executed sequentially, each on its separate execution context. This is very important, since it will allow each sfdx force:apex:execute command to be executed on its own Governor Limits.

    Optional Step 6: Execute multiple scripts for the same SObject

    You may ask: Why create multiple script files for the same SObject? Well, we just mentioned that each script file (*.apex) is executed by the Salesforce CLI having its own limits. There are a lot of limits, but when you insert records you’re going to hate a limit called “Maximum CPU time on the Salesforce servers”, which is 10,000 ms (that is 10 seconds). In practise this limit may not allow you to insert more than 200 records in a complex implementation (with triggers, many validation rules and so on) or even no more than 50 records!

    So, let’s suppose that you try to insert 2,000 opportunities but the insertion fails because of this limit. What is the process you should follow?

    1. Try to execute the script file (sfdx force:apex:execute -f .\opportunities.apex -u OrgAlias) using 100 records, i.e. FObject(‘Opportunity’, 100)
    2. If your execution is successful, double the records (200), if not set the number of records to half (50).
    3. Follow the same logic, increasing or decreasing the number of records, to find an optimal maximum value. Take into consideration that Salesforce doesn’t always need the same time to execute your script, so it’s better to set a value less than the absolute maximum.

    For example, suppose that you find you can insert up to 250 records. It’s safer to decrease this by 10% and set 225 records.

    • Because you want to insert 2,000 records, but each script file will insert 225 records, you need to create 9 script files. The first 8 files will insert 225 records each and the last one will insert 200 records (225 * 8 + 200 = 2,000 records).
    • The second line of each script file contains the seed number (FObject.seed = …). Here we have something new: we’re going to use different seed numbers for the same SObject. You may use whatever numbers you want or something like 1,2,3, etc. So, the first file will have FObject.seed = 1, the second FObject.seed = 2, etc. Why do we set different seed values? If we don’t, Forceea will create the same (but random) values in each execution for this SObject (same Close dates, same Amounts, etc), something we don’t want at all! Of course, you may omit the seed command if you don’t want to have the same values whenever you execute your scripts.
    • Another important note is this: if you use seed numbers you should always delete the previously created records, unless the same seed number is never used again. For example, if you insert the 2,000 records using seeds from 1 to 9, if you insert new records using a seed between 1 and 9, these new records will be the same as some already created records (something BAD). So, you must use a new seed number (e.g. 10) or delete the records you inserted before inserting new ones.
    • Each script file may have the same or different field definitions. It’s up to you to decide what kind of data you want.
    • And a last tip: if any of your definitions uses the serial command (e.g. serial type(number) … in the Opportunity.apex), first insert the command obj.createRecords(numberOfNextrecord); before the final line obj.insertRecords(true); where numberOfNextrecord is the number of the next record to be inserted. For example, in the first script file it will be createRecords(1), in the second createRecords(101), in the third createRecords(201), etc – under the assumption that you insert 100 records in each script file. If you follow this tip, all inserted records will have a continuous serial number!

    Conclusion

    Congratulations! You finally managed to reach at the end of this step-by-step guide. I hope I convinced you that Forceea with Salesforce CLI will make your life much easier. After you create the set of your script files, you can automate the creation of all your SObjects using a unique batch file. CLI will execute your batch files line by line, creating and inserting your records, with each execution with new Limits (which is fantastic!!)

    It will be great to have your feedback after your try to create a few scripts in a Sandbox or Developer org. And don’t hesitate to send an email with any questions or issues you may have.

[Salesforce / Apex] Forceea data factory framework: a new approach to an old problem

By

On April 5, 2018

In Post

This week’s guest post is a cool technical post about Apex Tests by Nikos Mitrakis.

Apex Tests are perceived as the hated necessity by most Salesforce developers: they are really important to keep high code quality levels but they are not the most fun part of coding with Salesforce.

I casually discovered Nikos through his GitHub page: I had a read at this Forceea Apex Test Framework, and I asked him to write a post to describe this cool piece of Salesforce tech.

Nikos has started his Salesforce journey in 2014. He started with Salesforce as a hobby (which still remains) and his wife continously threatens him with divorce if he doesn’t stop playing with his various Orgs all the time.
He has a Physics degree, a Microsoft SQL Server BI certification and a few Salesforce certifications. He likes writing code and discoverig how huge the Salesforce ecosystem is. He adores talking about his personal projects, he likes good sci-fi films, he’s been working with a Mac for more than a year, and he started the Salesforce Developer Group in Greece in 2016 (admittedly without much success).
Nikos is working as a Salesforce Developer at the Johnson & Johnson EMEA Development Center in Ireland. If he wasn’t a Salesforce Developer, he’d like to be a Developer in Salesforce.

What is a Data Factory?

Generally speaking, a Data Factory is a set of tools that generate and insert data. It’s very important when you develop a test method in an Apex Test Class, since you usually have to insert records to complete the testing. Using a Data Factory you make your life easier and your work more productive!

A Data Factory is also required when you want to test a report, a trigger/class or another process under Large Data Volumes (e.g. how fast is our report when there are 1 million records?), for User Acceptance testing (UAT), and for training – when you don’t want to create a Full Sandbox or when you want to create data which don’t exist in the production system yet.

What is Forceea and how I can install it?

Forceea (forˈsēa) is a data factory framework for Salesforce and an open source project (https://github.com/nmitrakis/Forceea) in GitHub. It’s a framework, because it gives the programmatic tools for a developer (or administrator, why not?) to easily generate, insert and delete SObject records.

“Yet another data factory?” you may say. Well, not at all!

But first let’s see how you install it.

When you download the files, your installation depends on your preference: classic metadata or DX? For the first option, you may want use the Ant (the build files are included) or the CLI, for example with

sfdx force:mdapi:deploy -u <OrgAlias> -d <folder> -w 5).

For DX, I suggest the CLI, using the command

 sfdx force:source:push -u <OrgAlias>

How can I generate the data I want?

OK, you have installed the framework. Now what?

I think the better way to understand how it works is by example: suppose you want to insert 100 Opportunity records and in your Org the Opportunity object has the additional required custom field MyField__c (which is a lookup to the custom object MyObject__c). Of course the Opportunity has its own required fields (like Stage).

The first step is to create an FObject:

FObject myObj = new FObject(Opportunity, 100);

Next, you declare the field definitions. A field definition uses a descriptive “language” to define what kind of data you want to create for this particular field. This language is called SDDL (Sample Data Definition Language). Don’t be afraid! It’s not a new programming language and it’s very easy to learn.

Let’s start with the first field you want to define, which is the Record Type (API name: RecordTypeId). Do you want your records to have a specific record type (let’s say BigDeal)? Then give this field definition:

myObj.setDefinition('RecordTypeId', 'static value(BigDeal)');

static here is the command and value is the parameter. The static command creates (what else?) static values, that is values which are the same in every record you create.

Do you want a random record type? Then you need the following definition:

obj.setDefinition('RecordTypeId', 'random type(picklist)');

The random command does exactly what you understood, it’s a really powerful command and you can even decide that every time you get the same (but random) data!

But wait! I hear you saying that the Record Type field is not a picklist field, and of course you’re right (it’s a Lookup)! Forceea handles this “special” field as a picklist field, making things easier for you. Say “Thank you 4ca”..

So, we move to our next field, which is the Close Date (API name CloseDate). This has a Date type, and let’s suppose you want it to have random Date values from this year:

myObj.setDefinition('CloseDate', 'random type(date) from(2018-01-01) to(2018-12-31)');

The type parameter is the boss here – it defines what kind of data you get. Give type(Boolean) and you get random True/False values. Give type(phone) and you have random phone numbers, etc

The Amount is a currency, so you decide to take random values from 100,000 to 1,000,000 and round these values to the nearest thousand. Difficult? No, it’s very easy!

myObj.setDefinition('Amount', 'random type(number) from(100000) to(1000000) scale(-3)');

Every opportunity that respect’s itself should have an Account, so you decide that the AccountId field will take an ID from any random Account, but you’ll get accounts without two specific Industry values. You asked for it, you get it:

myObj.setDefinition('AccountId', 'random lookup(Account) field(Industry) except(Banking,Chemicals) source(forceea)

I think the above definition needs some clarifications. The lookup(Account) instructs the framework to fetch records from the Account object and the field(Industry) except(…) to get only records for which the Industry field has any value except “Banking” and “Chemicals”.

But what is source(forceea)? This parameter defines the “source” of the lookup object’s records:

  • If it’s source(salesforce), the framework will query the Salesforce database and get the lookup records.
  • If it’s source(forceea) and the lookup object has been previously created (in the same context/transaction), it gets the records from there (which of course is much faster).
  • If it’s source(forceea) and the lookup object hasn’t been created (or the records have been inserted and deleted), it inserts some records of this lookup object.

Now that you have the opportunity account, don’t forget the opportunity Name (API name Name). Let’ say you want it to have the format <AccountName> – Opportunity for <Random text (with words) from 15 to 30 chars>

So, what is our definition? Well, we don’t have one, but 3 definitions to deliver this:

myObj.setDefinition('Name', 'copy field(AccountId) from(Account.Name)');

This definition gets the Name of the related account.

myObj.setDefinition('Name', 'static value(" – Opportunity for ")');

As you see, this is just a simple text value.

myObj.setDefinition('Name', 'random type(text) minlength(15) maxlength(30)');

And finally, we get our random text with “Latinoid” random words, like “Partem inermis ius impedit eam”

Keep in mind that with the random type(text) definition, the first character of the text is always a capital and the same word is never repeated twice in a row.

Your next field is Lead Source (which is a picklist field with the API name LeadSource), and here you want to have any declared picklist value except from “Partner” and “Other”:

myObj.setDefinition('LeadSource', 'random type(picklist) except(Partner, Other)');

If you wanted these two values only, you could give the definition:

myObj.setDefinition('LeadSource', 'random type(list) value(Partner, Other)');

or if you needed just a specific picklist value:

myObj.setDefinition('LeadSource', 'static value(Partner)');

Now you’ve finished with your field definitions, but remember that we haven’t defined all required fields (like Stage or myField__c). Of course you don’t have to worry about that, because the framework automatically identifies and defines all required fields. There are specific rules for setting the suitable field definition, but (without going to many details) we could briefly say that the field definition for a required field depends on the field data type, the SObject and the field name. So, after defining your fields, Forceea will define all required fields you didn’t define, like Stage or myField__c.

If you need to make some changes in the created records (for example to update a field with some complex calculations), you can get the opportunities with:

List<Opportunity> myRecords = (List<Opportunity>) myObj.getRecords();

Then do your changes, e.g.

for (Opportunity objRecord : myRecords) {
 objRecord.Amount = … // your changes here
}

and just insert the amended records with

myObj.insertRecords(true);

Forceea has many other field definitions, to create random real first and last names (with male and female names), real postal addresses (address, city, postal code, state, country), URLs, email addresses, phone numbers, Boolean values and Datetime values. And don’t forget that it supports BLOBs (e.g. to create email attachments).

It also provides definitions with the serial command (e.g. serial type(number) from(1) step(1.5) scale(2)) to create serial integer, decimal, currency, percentage, date and datetime values.

As you understand, there are many methods to

  • get or delete field definitions
  • create records from a specific serial number (very useful when you insert records with the serial command in different transactions)
  • insert records in groups (a valuable tool for your test methods)
  • define the Verbose mode (how many debug logs you get)
  • get the errors (yes, there is a detailed error system)

The framework handles field dependencies (dependent picklists) and it protects you from doing errors like defining the wrong definition for a specific field data type or defining multiple definitions for a field (e.g. Date) which can have only one.

How can I see the framework’s messages?

Forceea doesn’t have a UI (for the moment). The framework uses Debug Log to post its messages, which show

  • The errors during the process
  • The process milestones completed
  • The data of the first created records

Here is a sample output, just to get an idea of what this looks like (please note that the field definitions in this sample are not the field definitions you used previously).Keep in mind that you have many options to reduce the quantity of logs, for example to get almost nothing on a production system (which by the way is accomplished using a Custom Metadata type).

What’s next?

When you use Forceea in a test method, you really don’t need to create many records, so the 10,000ms CPU limit isn’t an actual problem. Of course you can use the framework to insert records in a sandbox or developer Org when you execute an Apex script in the Anonymous Windows, which of course has the same 10,000ms limit. But when you want to populate a sandbox with 1,000,000 of Accounts, this is another story..

This story is a major part of the next upcoming release (v1.3). The functionality is called Async, it can insert or delete many objects in one transaction, and when you insert objects or delete 1 object it’s surprisingly fast! I have inserted 500,000 of a custom object (without any validation rules, triggers, processes, etc) in 40s! Of course its speed mainly depends on the triggers, etc you have in your actual Org, which may slow down the process significantly (that’s life..)

Anything else?

Yes! I invite everyone to have a look at the detailed User Guide (http://bit.ly/Forceea12_UserGuide) and try the framework.

And I’d like to say a big THANKS to Enrico for hosting this post.

Powered by WordPress & Theme by Anders Norén