Tag: Data Factory

• Salesforce Developer at Johnson & Johnson EMEA Development Centre (EDC)
• Started his Salesforce journey in 2014
• Has passed 9 certifications, including Certified Application Architect
• Holds a Physics degree
• Leader of Limerick, Ireland Developer Group
• Married since 1994, has a daughter
• Loves watching sci-fi movies and good comedies
• Lives in Limerick, Ireland

All posts by Nikos

So, you are an awesome Administrator, Developer, Consultant or Architect and you have your Scratch org, Developer Sandbox or Developer edition org ready for development, testing, training or a demo. But your org doesn’t have any data ☹, or it doesn’t have the data you want. What can you do?

Well, you have the following options:

1. Use an ETL tool to move data from another Org or get data from CSV files.
2. Use Salesforce CLI (Command Line Interface) using the commands sfdx force:data:tree:export and force:data:tree:import to export/ data from another Org and import the data (in JSON format).
3. Develop a custom solution (Apex) to insert the records you need programmatically.
4. Use a Data Factory to insert the required data.

The first (ETL) and second (CLI) solutions don’t need development skills, they may not be very difficult to configure (using CLI should be easier), and they let you populate your org with a specific set of records – if you really want this. The problem is that data requirements change very fast and if you have to continuously create manually the records you need in one org and then copy them to your org, this process will eventually be slow and time-consuming.

The third (Apex) solution gives you the maximum flexibility, but it’s time-consuming as well, not to mention the high level of technical knowledge it requires.

Now let’s see the fourth solution, the Data Factory. Using a Data Factory, you won’t have to write complex code or have advanced knowledge of an ETL tool. You’ll be able to easily generate and insert the records you need for any SObject and easily modify the process to adopt any new data requirements. Sounds interesting?

Stay with me as we’re going to implement such a data population project using Forceea (forˈsēa) data factory framework – this framework is an open source GitHub project (https://github.com/nmitrakis/Forceea). Keep in mind that the framework is perfect for a developer who wants to create records for the test methods, but that’s another story.

In this step-by-step guide we’ll assume that you have no programming knowledge! (you may be even a power Salesforce user). Ready to start? (Take a deep breath…)

Step 0: Prerequisites – Prepare your org

• For Classic orgs: You have created a Sandbox or a Developer org.
• For Salesforce DX Scratch orgs: you have created your scratch org.

For both Classic and DX, after the org creation you should have deployed your project’s metadata and set your org ready for the final data population.

Step 1: Install the Salesforce CLI

If you haven’t installed the Salesforce CLI, go to https://developer.salesforce.com/tools/sfdxcli, download the CLI for your system and install it. Accept the default settings and after a while your Salesforce CLI will be ready to run.

Step 2: Authorize your org

To use CLI, you need to authorize your org. To do this, open the Command Prompt window and execute the following command:

sfdx force:auth:web:login -r https://login.salesforce.com -a OrgAlias

Some comments here:

• If you’re authorizing a sandbox, the instance URL will be https://test.salesforce.com.
• You should replace OrgAlias with the alias of your org (choose one, e.g. sf1).

To verify that everything is fine, log into your org using the command

sfdx force:org:open -u OrgAlias

Step 3: Install Forceea

There are two ways to install Forceea.

Method 1: Go to https://github.com/nmitrakis/Forceea and click the following button:

You’ll be re-directed to the page:

• If you install in a sandbox, select Deploy to: Sandbox. Otherwise, accept the default selection (Production/Developer).
• Click the Login to Salesforce button (on the right corner) and give your credentials.
• Then click Allow to give the tool the access to your org. You are redirected to a new page where the From GitHub Repository section lists the details of Forceea URL and the To Salesforce Org section the details of the target org. Click the Deploy button (on the right corner).
• After a few seconds you’ll see the message Deployment Complete at the bottom of your page.

Method 2: Go to the GitHub repository https://github.com/nmitrakis/Forceea and click Clone or Download.

Click Download ZIP. After a few seconds the Forceea-master.zip will have been downloaded. Unzip the zip file and you will find the metadata folder, which contains

• the src folder, with a folder with each metadata component and the package.xml
• the dx folder, with the Salesforce DX components
• a build.properties file
• a build.xml file

The easiest way to regularly use Forceea is to include its metadata into your Repository (either classic or DX metadata). Then you can deploy it with your project’s metadata.

Step 4: Create the Forceea scripts

What are Forceea scripts?

Forceea uses a language (Sample Data Definition Language – SDDL) to describe the “nature” of the required data. A script is a set of commands. Let’s see an example, which creates and inserts opportunities:

FObject obj = new FObject('Opportunity', 100);
obj.setDefinition('Name', 'static value(Opp-)');
obj.setDefinition('Name', 'serial type(number) from(1) step(1) scale(0)');
obj.setDefinition('AccountId', 'random lookup(Account) field(Industry) value(Chemicals,Construction) source(salesforce)');
obj.setDefinition('Amount', 'random type(number) from(1000000) to(10000000) scale(-3)');
obj.setDefinition('StageName', 'random type(picklist)');
obj.setDefinition('CloseDate', 'random type(date) from(2017-01-01) to(2017-12-31)');
obj.setDefinition('Forceea__c', 'static value(xJio9!23)'); // ** create this field **
obj.insertRecords(true);

If the above commands look like Apex statements, you’re right, they are Apex statements, but you may forget it for the moment! The above script will insert 100 Opportunity records with:

• Name: Opp-1, Opp-2, .., Opp-100
• Account: any account for which the Industry is “Chemicals” or “Construction”
• Amount: a random integer from 1M to 10M, round to 1000 (e.g. 9,252,000)
• Stage: any picklist value
• Close date: any date from 1 Jan. 2017 to 31 Dec. 2017

A field definition begins with obj.setDefinition and has two text parameters:

• The field API name, e.g. Amount or AccountId
• The SDDL definition, e.g. random type(number) from(1000000) to(10000000) scale(-3) or random type(picklist).

Finally, we insert the created records using obj.insertRecords(true). The parameter true means that it is all or nothing, so if any error happens, no records will be inserted.

The framework has many cool features:

• Creates records for standard or custom objects, for any standard or custom field.
• Automatically defines required fields.
• Can create data for fields of every data type: Integer, Currency, Double, Date, Datetime, Time, Boolean,
  String, TextArea, Percent, Reference, Email, Phone, URL, Base64, Picklist and MultiPicklist.
• Handles record types and field dependencies (dependent picklists).
• Supports record groups for inserting and deleting records.
• Validates the definitions based on the field data type.
• Can be used for in test methods or for populating Salesforce orgs with sample data for demos and User
  Acceptance Testing (UAT).
• Has an extended error messaging system.

You can get a lot of script examples from http://bit.ly/Forceea131_Examples and download the User Guide from http://bit.ly/Forceea131_UserGuide.

Write your own scripts

Now you know what is a Forceea script, let’s see how you can write the scripts for your specific needs. To do this, use the following template:

FObject obj = new FObject('SObject_API_Name', Number_of_records);
FObject.seed = 1;
<field definitions>
obj.insertRecords(true);

where are one or more SDDL field definitions, like the previous definitions we used to create a script for Opportunity records.

The seed is very important when you want to create the same random records every time. For example, when you populate a Scratch org with sample data, we may want each developer to have the same (but random) set of records. Forceea uses its own engine to create random data (generally called Pseudo-random Number generator – PRNG), which you can control.

Now, let’s suppose you want to populate your org with Account and Opportunity records. This means that first you must create records for accounts and then for opportunities (an opportunity is related to an account, that’s why we insert accounts first).

Accounts

Open your favourite text editor (I use VSCode) and create a script using the above template. Your script may look like:

FObject obj = new FObject('Account', 100);
FObject.seed = 1;
obj.setDefinition('RecordTypeId', 'static value(BigAccount)');
obj.setDefinition('Name', 'static value("Company ")');
obj.setDefinition('Name', 'serial type(number) from(100) step(1) scale(0)');
obj.setDefinition('NumberOfEmployees', 'random type(number) from(10) to(1000) scale(-1)');
obj.setDefinition('AnnualRevenue', 'random type(number) from(10000000) to(100000000) scale(3)');
obj.setDefinition('Rating', 'random type(picklist) except(Hot)');
obj.setDefinition('Phone', 'random type(phone) format("(30) 210 dD-00-DD")');
obj.setDefinition('Industry', 'random type(picklist)');
obj.setDefinition('Type', 'random type(list) value(Prospect, Customer - Direct, Customer - Channel)');
obj.setDefinition('Site', 'random type(url)');
obj.setDefinition('ShippingStreet', 'random type(street) group(shipping)');
obj.setDefinition('ShippingPostalCode', 'random type(postalcode) group(shipping)');
obj.setDefinition('ShippingCity', 'random type(city) group(shipping)');
obj.setDefinition('ShippingState', 'random type(state) group(shipping)');
obj.setDefinition('ShippingCountry', 'random type(country) group(shipping)');
obj.insertRecords(true);

Even though we don’t have the space here to describe every SDDL definition of the above script, I’d like to mention the first field definition for field RecordTypeId (if you use record types, modify the definition with the API name of your record type instead of BigAccount, otherwise delete this line) and the address definitions ShippingStreet, ShippingPostalCode, etc (where you get real addresses from United States).

You may think that these definitions are difficult, but most of them are obvious and easy to understand. So, back to our script for the Account SObject, give it the name Accounts.apex and save it into your preferred folder (let’s call it ForceeaScripts).

Opportunities

Now it’s time to create the Opportunities.apex file with the script for the Opportunity SObject. Follow the previous procedure and write a script like the example I gave you above.

Keep in mind that Forceea will try to automatically find the required fields and generate a field definition for them but, as a rule of thumb, you should always create your definitions for all required fields.

Record Deletion

When you create records, you probably want to repeat this procedure again, and replace the existing set of data with another set with different or the same field values. To do this, you probably want to delete the existing inserted records before you insert the new ones. This is the responsibility of the Record Deletion files.

First, we’ll delete the opportunities we created. Follow these steps to configure it:

• Create a new text file, call it delete-opportunities.apex and save it in the ForceeaScripts folder.
• Write this script in the file:

  String flag = 'xJio9!23';
  List<Opportunity> records = [SELECT Id FROM Opportunity WHERE Forceea__c = :flag LIMIT 5000];
  delete records;
  

You’re right! At first sight these commands may look difficult if you are not a developer, but they aren’t really hard to understand if you follow these instructions:

• Line 1: You may select your own (random) value, like xJio9!23 – but don’t forget to enclose it in single quotation marks
• Line 2: Find a filed you don’t use, which has a Text data type or create your own text field and call it Forceea__c. Here I use the latter, because I couldn’t find any available (not used) standard field on the Opportunity SObject.

Now go to the opportunities.apex file and add the following line before the command obj.insertRecords(true); :

obj.setDefinition(Forceea__c', 'static value(xJio9!23)');

As you can see, we set the field definition for the field Forceea to be our static value xJio9!23. Do the same for the Account SObject, creating the file delete-accounts.apex with the following commands:

String flag = 'xJio9!23';
List<Account> records = [SELECT Id FROM Account WHERE Tradestyle = :flag LIMIT 5000];
delete records;

In Line 2, I suppose the TradeStyle text field is not used in your org, so you may use it here. Else, you should create a new field as previously.

Go to the accounts.apex file and add the following line before the command obj.insertRecords(true); :

obj.setDefinition('TradeStyle', 'static value(xJio9!23)');

Be careful! The above scripts will delete up to 5,000 records, so if you have more records you have 2 options:

• Increase LIMIT 5000 up to 10000 (e.g. LIMIT 8000)
• Execute the delete-accounts.apex again (you may use it as many times as you want)

That’s it! Now you have 4 files into your ForceeaScript folder:

• account.apex
• opportunities.apex
• delete-accounts.apex
• delete-opportunities.apex

For your convenience, I have created these files in a zipped file, which you can download from http://bit.ly/ForceeaWithCLI.

You may use these files for your training. I suppose you should modify them if you want to use them in a production system (with specific requirements for the field values), but you’ll probably need no modifications if you execute them in an out-of-the-box Developer/Scratch org.

Step 5: Execute your scripts

The last step is to execute the scripts and insert the created records. Create a new text file, call it execute-scripts.bat and save it in the ForceeaScripts folder (I included it in the above zipped file).

In this file you’re going to insert the CLI commands which execute the *.apex files you created previously.

If you use Windows, your execute-scripts.bat file will look like:

sfdx force:apex:execute -f .\delete-opportunities.apex -u OrgAlias
sfdx force:apex:execute -f .\delete-accounts.apex -u OrgAlias
sfdx force:apex:execute -f .\accounts.apex -u OrgAlias
sfdx force:apex:execute -f .\opportunities.apex -u OrgAlias

where orgAlias is the alias for your org you defined in Step 2.

If you’re using a Mac, each command may look like:

sfdx force:apex:execute -f /Users/Username/Documents/ForceeaScripts/accounts.apex -u OrgAlias

When you write your Forceea scripts (e.g. your script in the accounts.apex file), you may have errors to correct. For example, you may have misspelled a field’s API name or your SDDL definition may have a syntax error (a very common source of errors is the effect of the Validation Rules and Triggers). Fortunately, the framework has an extensive debug log, with detailed error messages (see the User Guide, pages 44 – 50). You can activate these debug logs, if you insert the command obj.setVerbose(‘info’) or obj.setVerbose(‘debug’) after the first command of your accounts.apex or opportunities.apex script files.

I suggest you initially execute your script files one by one.

• If you already have inserted records, first execute the two delete-object files. Copy the “delete” commands from your execute-scripts.bat to your Command Prompt window and execute them. Then verify that records have been deleted successfully.
• Copy the “insert” commands from your execute-scripts.bat file to your Command Prompt window and execute them one by one, verifying that records have been inserted successfully.
• When you have verified that everything works as expected, you may just execute your batch file in the Command Prompt window: > execute-scripts.bat (Window users)

If you are a Mac user, you could use a similar batch execution or just simply copy all commands of the execute-scripts file to the Command Prompt window and execute them at once.

Because of this process, all lines of the execute-scripts file will be executed sequentially, each on its separate execution context. This is very important, since it will allow each sfdx force:apex:execute command to be executed on its own Governor Limits.

Optional Step 6: Execute multiple scripts for the same SObject

You may ask: Why create multiple script files for the same SObject? Well, we just mentioned that each script file (*.apex) is executed by the Salesforce CLI having its own limits. There are a lot of limits, but when you insert records you’re going to hate a limit called “Maximum CPU time on the Salesforce servers”, which is 10,000 ms (that is 10 seconds). In practise this limit may not allow you to insert more than 200 records in a complex implementation (with triggers, many validation rules and so on) or even no more than 50 records!

So, let’s suppose that you try to insert 2,000 opportunities but the insertion fails because of this limit. What is the process you should follow?

1. Try to execute the script file (sfdx force:apex:execute -f .\opportunities.apex -u OrgAlias) using 100 records, i.e. FObject(‘Opportunity’, 100)
2. If your execution is successful, double the records (200), if not set the number of records to half (50).
3. Follow the same logic, increasing or decreasing the number of records, to find an optimal maximum value. Take into consideration that Salesforce doesn’t always need the same time to execute your script, so it’s better to set a value less than the absolute maximum.

For example, suppose that you find you can insert up to 250 records. It’s safer to decrease this by 10% and set 225 records.

• Because you want to insert 2,000 records, but each script file will insert 225 records, you need to create 9 script files. The first 8 files will insert 225 records each and the last one will insert 200 records (225 * 8 + 200 = 2,000 records).
• The second line of each script file contains the seed number (FObject.seed = …). Here we have something new: we’re going to use different seed numbers for the same SObject. You may use whatever numbers you want or something like 1,2,3, etc. So, the first file will have FObject.seed = 1, the second FObject.seed = 2, etc. Why do we set different seed values? If we don’t, Forceea will create the same (but random) values in each execution for this SObject (same Close dates, same Amounts, etc), something we don’t want at all! Of course, you may omit the seed command if you don’t want to have the same values whenever you execute your scripts.
• Another important note is this: if you use seed numbers you should always delete the previously created records, unless the same seed number is never used again. For example, if you insert the 2,000 records using seeds from 1 to 9, if you insert new records using a seed between 1 and 9, these new records will be the same as some already created records (something BAD). So, you must use a new seed number (e.g. 10) or delete the records you inserted before inserting new ones.
• Each script file may have the same or different field definitions. It’s up to you to decide what kind of data you want.
• And a last tip: if any of your definitions uses the serial command (e.g. serial type(number) … in the Opportunity.apex), first insert the command obj.createRecords(numberOfNextrecord); before the final line obj.insertRecords(true); where numberOfNextrecord is the number of the next record to be inserted. For example, in the first script file it will be createRecords(1), in the second createRecords(101), in the third createRecords(201), etc – under the assumption that you insert 100 records in each script file. If you follow this tip, all inserted records will have a continuous serial number!

Conclusion

Congratulations! You finally managed to reach at the end of this step-by-step guide. I hope I convinced you that Forceea with Salesforce CLI will make your life much easier. After you create the set of your script files, you can automate the creation of all your SObjects using a unique batch file. CLI will execute your batch files line by line, creating and inserting your records, with each execution with new Limits (which is fantastic!!)

It will be great to have your feedback after your try to create a few scripts in a Sandbox or Developer org. And don’t hesitate to send an email with any questions or issues you may have.