Custom data providers

Creating an Enterprise Data Provider Script For JSON data

JSON (JavaScript Object Notation) is a lightweight data interchange format that is easy for humans to read and write, and easy for machines to parse and generate. It is widely used for transmitting data between a server and a web application, as well as between various software components. JSON represents structured data using a collection of key-value pairs, arrays, and nested objects, making it highly flexible and versatile.

Despite its simplicity, JSON does not inherently organize data into records and field sets like you might find in a relational database. Instead, JSON structures data in a hierarchical manner, which can be deeply nested. This makes it an excellent choice for applications that require a flexible schema or need to handle complex data structures.

To leverage JSON data within EasyCatalog, you can transform it into a structure that EasyCatalog can import. This is typically done through a script that breaks down each JSON object into individual records and represents the keys within each object as fields. This transformation allows EasyCatalog to handle the JSON data effectively, treating each object as a separate record with fields corresponding to the JSON keys.

It’s important to note that this transformation process depends on the specific structure of your JSON data and the requirements of your application. You may need to customize the script to handle various data types, nested objects, and arrays to ensure a seamless integration with EasyCatalog.

In summary, JSON is a crucial format for structured data exchange due to its readability, flexibility, and ease of use. By converting JSON data into a format compatible with EasyCatalog, you can harness its power for a wide range of applications, from web development to data processing and beyond.

Here’s a simple script designed to handle a simply structured JSON File (Uses functions only available in the 2024 or later version of EasyCatalog).

---------------------------------------------------------------------------------------------------------
-- JSON 2024 Custom Data Provider.
--
-- Only works in the 2024 version of EasyCatalog
--
-- Key functions are called during data retrieval and update. The script is initialised and called on demand, 
-- as such it is statless and relies on the 'configration' settings passed form call to call to represent
-- the data sources current settings.
--
-- Data Source Creation
-- During creation of a new data source the call sequence is as follows:
-- 1. Initialize() - returns a table of default settings, which are [ersisted in the data sources datasource.xml file.
-- 2. ConfigureUI(config) presents configuration settings to the user to allows modification (not called on InDesign Server)
-- 3. Synchronize(datasource, config) - called after configuration to retreive a complete RECORDSET
--
-- Data Source Updates
-- SaveRecords(config, datasource, records) - is called when the user selects "Update Data Source". 
-- this is responisble for passing updates back and clearing the update flag.
--
---------------------------------------------------------------------------------------------------------
--
--  Oct 2023    Created         v1.0.0
--
---------------------------------------------------------------------------------------------------------

--[[
Example structure of JSON this sample is designed to load:
{
  "products": [
    {
      "id": "12345",
      "name": "Product A",
      "description": "This is Product A, a high-quality product.",
      "price": 19.99,
      "category": "Electronics",
      "stock": 50
    },
    {
      "id": "67890",
      "name": "Product B",
      "description": "Product B is a versatile and popular item.",
      "price": 29.99,
      "category": "Clothing",
      "stock": 100
    },
    {
      "id": "54321",
      "name": "Product C",
      "description": "Product C is a premium accessory.",
      "price": 39.99,
      "category": "Accessories",
      "stock": 25
    }
  ]
}
]];


-- This specifies the JSONPath to each 'record' in the resulting data 
local record_JSONPath = "$.products[*]"

-- This specifies fields to extract from each record 
local fields_with_options = {
    { name = "id", jsonpath = "$.id", key = "true"},
    { name = "name", jsonpath = "$.name"},
    { name = "description", jsonpath = "$.description"},
    { name = "price", jsonpath = "$.price"},
    { name = "stock", jsonpath = "$.stock"}
};

------------------------------------------------------------------------------------------------------------------------
-- load_file
------------------------------------------------------------------------------------------------------------------------
function load_file(path)
  local file = io.open(path, "r");
  if not file then return nil end
  local content = file:read "*a" -- *a or *all reads the whole file
  file:close();
  return content
end

------------------------------------------------------------------------------------------------------------------------
-- getActualScriptVersion
------------------------------------------------------------------------------------------------------------------------
function getActualScriptVersion()
  local info =  GetInfo();
  return 'v' .. info.version;
end


---------------------------------------------------------------------------------------------------------
-- Initialize - Define the data sources default settings. 
-- Returns a table of settings.
---------------------------------------------------------------------------------------------------------
function Initialize()

  config = {}
  config.name = "JSON 2024";
  config.initialized  = false;
  config.JSONfile = '';
  config.version = getActualScriptVersion();            -- future usage, new.15.Dec.2021, started at 'v4.8.6'
  return config;
end




---------------------------------------------------------------------------------------------------------
-- validatemethod 
---------------------------------------------------------------------------------------------------------
function validatemethod(dialog)

  local name = dialog:getwidget("Name").value;
  if name == "" then
    DIALOG.alert("please enter a valid name.");
    return "Name";
  end

  local file = dialog:getwidget("edit_file").value;
  if file == "" then
    DIALOG.alert("please select a JSON file to process.");
    return "edit_file";
  end

  return "";
end

---------------------------------------------------------------------------------------------------------
-- setDialogWidget
---------------------------------------------------------------------------------------------------------
function setDialogWidget(pConfigdialog, pName, pValue)
  local widget = pConfigdialog:getwidget(pName);
  widget.content = pValue;
  pConfigdialog:setwidget(widget);
end

---------------------------------------------------------------------------------------------------------
-- get_JSON_file_location
--
---------------------------------------------------------------------------------------------------------
function get_JSON_file_location()
  -- ask user for JSON file location
  local done, path = DIALOG.choosefile("Choose JSON file");
  return done, path;
end

---------------------------------------------------------------------------------------------------------
-- chooseButtonAction
---------------------------------------------------------------------------------------------------------
chooseButtonAction = function(configdialog) 
  local done, path = get_JSON_file_location();     -- JSON file location
  if (done) then
    actionConfig.JSONfile = path;
    setDialogWidget(configdialog, "edit_file", actionConfig.JSONfile);
  end
end



---------------------------------------------------------------------------------------------------------
-- ConfigureUI
--
---------------------------------------------------------------------------------------------------------
function ConfigureUI(config)
  
  local enable_name = false;

  actionConfig = config;

  if config.initialized == false then   
      enable_name = true;
  end

  local title_new = "New JSON 2024 Data Source";
  local title_edit = "JSON 2024 Data Source";
  local the_title = title_new;

  if enable_name == false then
    the_title = title_edit;
  end
  configdialog = DIALOG.new( { title = the_title, validate = validatemethod } );
  
  local name_top            = 20;
  local file_top            = 20+(25*1);
  local choose_button_top   = 20+(25*2);
  local button_top          = 20+(25*3)+10;
  local widget_left         = 90;            
  local static_right        = widget_left;    
  local widget_right         = 500;            
  local choose_button_width  = 100;
  local choose_button_left   = widget_right - choose_button_width;

  configdialog:addwidget(
  {

      { type = "statictext", title = "Name:", align = "left", left = 20,  top = name_top, right = static_right, height = 20	 	},
      { type = "editbox",  id = "Name", left = widget_left, top = name_top,  right = widget_right,  height = 20,  content = config.name, enable = enable_name	},     
      { type = "statictext", title = "File:", align = "left", left = 20,  top = file_top, right = static_right, height = 20	 	},
      { type = "editbox",  id = "edit_file", left = widget_left, top = file_top,  right = widget_right,  height = 20,  content = config.JSONfile, enable = true	},
      { type = "button", id = "edit_file_choose", title = "Choose", left = choose_button_left,  top = choose_button_top,  width = choose_button_width,  heigth = 20,  onchange = chooseButtonAction},
      { type = "cancelbutton", title = "Cancel", id = "cancel", left = 270, top = button_top, width = 80, heigth = 20,  },
      { type = "okbutton", title = "Ok", id = "ok", left = 270+80+20, top = button_top, width = 80, heigth = 20 },

  }
  );

  if configdialog:open() then
	  config.name       = configdialog:getwidget("Name").content;
	  config.JSONfile   = configdialog:getwidget("edit_file").content;
    return config;
  end
  
end


---------------------------------------------------------------------------------------------------------
-- Synchronize
-- config  : Configuration parameters (in/out).  If changed, will update datasource.xml 
-- datasource : Data source object
-- Returns :  new RECORDSET or nil if canceled
---------------------------------------------------------------------------------------------------------
function Synchronize(config, datasource)
  config.initialized = true;
  return RECORDSET.new(fields_with_options, create_records_from_json(config));
end

---------------------------------------------------------------------------------------------------------
-- SaveRecords
-- Update changes back to the data source 
-- config  : Configuration parameters (in/out)
---------------------------------------------------------------------------------------------------------
function SaveRecords(config, datasource, records) 

  for i=1,records:size() do
    record = records:getrecord(i);
    for x=1, record:size() do
      field = record:field(x);
      if field:getupdatestate() & 2 == 2 then
        field:setupdatestate(0);
      end
    end
  end
end


---------------------------------------------------------------------------------------------------------
-- GetReleaseNotes
--
--  This is called when the ‘info’ button is selected associated with a data provider row on the   
--  Manage Enterprise Data Provider dialog.
---------------------------------------------------------------------------------------------------------
function GetReleaseNotes()

  local body = "";

  body = body .. "<h1>JSON 2024 Release Notes</h1>";

  body = body .. "<h3>v1.0.0</h3>";
  body = body .. "<ul>";
  body = body .. "<li>A sample script using JSONPath to parse JSON data</li>";
  body = body .. "</ul>";

  c = {
    title = "JSON 2024 Release Notes",
    body = body,
  };
  
  return c;
end

---------------------------------------------------------------------------------------------------------
-- GetInfo
---------------------------------------------------------------------------------------------------------
function GetInfo()
  c = {
      url        = "https://www.65bit.com/docs/creating-custom-json-data-providers/",
      name       = "JSON 2024 File Sample",
      version    = "1.0.0",
 };
 return c;
end

---------------------------------------------------------------------------------------------------------
-- Given a URI, return true if the download should be handled by this script 
---------------------------------------------------------------------------------------------------------
function CanResolveAssetURI(field, uri) 
  return false;
end

---------------------------------------------------------------------------------------------------------
-- Turns a URI into a fully qualified location, from which the storage location is determined
---------------------------------------------------------------------------------------------------------
function ResolveAssetURI(config, uri) 
  return uri;
end

---------------------------------------------------------------------------------------------------------
-- Download the asset to the local cache
-- location : result of 'ResolveAssetURI' 
-- filepath : locale cache full path
---------------------------------------------------------------------------------------------------------
function GetAsset(config, location, filepath) 
  return false;
end

---------------------------------------------------------------------------------------------------------
-- Returns a table of records taken from the JSON with the fields as specified in 'fields_with_options' 
-- config : Configuration table with file location 
---------------------------------------------------------------------------------------------------------
function create_records_from_json(config)
    json = load_file(config.JSONfile);
    if json == nil or json == "" then
        error("JSON empty or cannot be loaded");
    end
    records = {}

    -- Use the records JSONPath to create a table of JSON data
    table_of_records, err = jsonarraytotable(json, record_JSONPath);
    if table_of_records == nil then
        error(err);
    end

    for r = 1, #table_of_records do
        record_json = table_of_records[r];
        if record_json ~= "" and record_json ~= nil then 
            r = {}
            for i = 1, #fields_with_options do
                r[fields_with_options[i].name] = processjson('jsonpath', record_json, fields_with_options[i].jsonpath);
            end
            table.insert(records, r);
        end
    end
    return records;
end