editing in loffice xml flat format

views/implementation.md

@@ -1,14 +1,21 @@
 # Implementation
 
-## Behavior Driven Development
+This section describes the salient implementation details of the Zencode DSL, the smart-rule language for DECODE, tailored on its use-cases and based on the Zenroom controlled execution environment (VM). Implementation details refer only to Zencode and not to how Zenroom is implemented, since the latter is already covered in other documents.
 
-In Behavior Driven Development (BDD), the important role of software integration and unit tests is extended to serve both the purposes of designing the human-machine interaction flow (user journey in UX terms) and of laying down a common ground for interaction between designers and stakeholders. In this Agile software development methodology the software testing suite is based on natural language units that grant a common understanding for all participants and observers.
+The implementation section contains three parts explaining:
+- the language model inherited by Behaviour Driven Development
+- the data validation model based on Schema Validation
+- the implementation of implicit certificates
 
-To implement BDD the first step is that of mapping a series of combinable, cascading sentences to actual source code; this implementation is usually done manually by programmers that have knowledge of the higher level application protocol interface (API) that grants communication between the backend and the frontend of a software application. The BDD implementation can then be seen as an alternative frontend whose purpose is that of lowering the distance between expression and execution by means of utterances expressed in human language.
+## Behaviour Driven Development
 
-Far from giving an exhaustive description of BDD implementations and characteristics, this brief chapter intends to summarize the features of this approach where they specifically apply to the development goals of Zencode (previously stated) and the solution provided.
+In Behaviour Driven Development (BDD), the important role of software integration and unit tests is extended to serve both the purposes of designing the human-machine interaction flow (user journey in UX terms) and of laying down a common ground for interaction between designers and stakeholders. In this Agile software development methodology the software testing suite is based on natural language units that grant a common understanding for all participants and observers.
 
-Referring to the Cucumber implementation of BDD, arguably the most popular in use by the industry to day and de-facto standard, the grammar of utterances is very simple and definible as a "cascading" flow indeed, since the fixed sequence of lines can follow only one fixed order:
+To implement BDD the first step is that of mapping a series of interconnected cascading sentences to actual source code; this implementation is usually done manually by programmers that have knowledge of the higher level application protocol interface (API) that grants communication between the backend and the frontend of a software application. The BDD implementation can then be seen as an alternative frontend whose purpose is that of lowering the distance between expression and execution by means of utterances expressed in human language.
+
+Far from giving an exhaustive description of BDD implementations and characteristics, this brief chapter intends to summarise the features of this approach where they specifically apply to the development goals of Zencode (previously stated) and the solution provided.
+
+Referring to the Cucumber implementation of BDD, arguably the most popular in use by the industry to day and factual standard, the grammar of utterances is very simple and definable as a "cascading" flow indeed, since the fixed sequence of lines can follow only one fixed order:
 
 - Given
  - and*
@@ -24,16 +31,16 @@ The underlying parser acts upon a positive, unique and so far non-flexible match
 
 Brief examples of this implementation follow:
 
-```lua
+```
 Given("I introduce myself as ''", function(name) whoami = name end)
 Given("I am known as ''",         function(name) whoami = name end)
 ```
 
-The above definition of two lines possibly occurring within the utterances in Zencode are demonstrating how a state "who am I" basically my own name can be set using two different phrases, leading to the execution of the same function which basically operates a simple assignment to the variable `whoami`. This simple demonstration is a hint to the fact that multiple patterns can be defined also in different ways, making the Zencode DSL implementation very easy to translate across different spoken languages as well contextualized within specific idiolects adopted by humans.
+The above definition of two lines possibly occurring within the utterances in Zencode are demonstrating how a state "who am I" basically my own name can be set using two different phrases, leading to the execution of the same function which basically operates a simple assignment to the variable *whoami*. This simple demonstration is a hint to the fact that multiple patterns can be defined also in different ways, making the Zencode DSL implementation very easy to translate across different spoken languages as well contextualised within specific idiolects adopted by humans.
 
 Furthermore, another example of implementation:
 
-```lua
+```
 Given("that '' declares to be ''",function(who, decl)
 		 -- declaration
 		 if not declared then declared = decl
@@ -48,11 +55,19 @@ Given("declares also to be ''", function(decl)
 end)
 ```
 
-Shows how is possible to accept multiple variables and process them through more complex transformations that also contemplate the concatenation of contents to previous states. States are in fact permanent within the scope of the execution of a single utterance and will be modified in the same deterministic order by which they are expressed across lines. What is also visible within this example implementation, which we intend to facilitate by customisations made by people who have a simple knowledge of Zenroom's API and LUA scripting, is that the `ZEN.` namespace makes available a number of utility functions to easily check states (asserts) and propagate meaningful error messages that are then part of a traceback output given to the calling application (host) on occurrence of an error.
+Shows how is possible to accept multiple variables and process them through more complex transformations that also contemplate the concatenation of contents to previous states. States are in fact permanent within the scope of the execution of a single utterance and will be modified in the same deterministic order by which they are expressed across lines. What is also visible within this example implementation, which we intend to facilitate by customisation made by people who have a simple knowledge of Zenroom's API and LUA scripting, is that the 'ZEN.' namespace makes available a number of utility functions to easily check states (asserts) and propagate meaningful error messages that are then part of a backtrace output given to the calling application (host) on occurrence of an error.
 
-The full implementation of Zencode available at the time of publishing this document is inside the sourcecode files `zenroom/src/lua/zencode_*` and is relatively easy to maintain for the pilots analysed in our project, as well easy to extend to more usecases. At the dawn of piloting sessions, due to the lack of actual feedback so far given in field trials, this implementation does not addresses specific schemes beyond a simple Diffie-Helman asymmetric key encryption (AES-GCM) and an even simplier symmetric encryption of ciphertext by means of a PIN and KDF transformations. On top of that, perhaps the most complex implementation of Zencode so far is the "implicit certificate" crypto scheme (Qu-Vanstone, ECQV) still limited to first order curve transformations, which applies widely to pilots requiring simple certification schemes and is illustrated in more detail in the following chapters[^ecqv].
+The full implementation of Zencode available at the time of publishing this document is inside the source-code files 'zenroom/src/lua/zencode_*' and is relatively easy to maintain for the pilots analysed in our project, as well easy to extend to more use-cases. The current implementation addresses specific schemes that useful to the pilots in DECODE, while contemplating future extension:
+- Simple symmetric encryption of ciphertext by means of a PIN and KDF transformations (pilot: Amsterdam Register)
+- Diffie-Hellman asymmetric key encryption (AES-GCM) (pilot: Making Sense IoT)
+- Blind-sign credentials for unlinkable selective attribute revelations[^coconut] (pilot: DECIDIM and Gebied Online)
+In addition there is also the implementation of an "implicit certificate" crypto scheme (Qu-Vanstone, ECQV) that is limited to first order curve transformations, which may apply to pilots requiring simple certification schemes[^ecqv]. All the implementations are illustrated in more detail in the following chapters.
 
-[^ecqv] It is important to note that while the ECQV scheme was not examined by other partners in our project, it has been choosen for its stable role in the industry and for its augmented complexity within an approachable implementation, complexity which could better inform the Zencode implementation. Without that complexity and without implementation feedback by other partners, it wouldn't have been possible to work on Zencode and bring it to what it is today, since both the Petition contract and the Coconut implementation in Zenroom are not available as of today and need to be completed in a later stage of the DECODE project.
+
+[^coconut] This implementation refers to work on the Coconut credential system (Sonnino et. al, 2018) designed after specific needs in DECODE's pilots. It does not implement, however, the threshold issuance part, which is only required in the scenario of a fully open blockchain implementation, which is still work in progress.
+
+
+[^ecqv] It is important to note that while the ECQV scheme was not examined by other partners in our project, it has been choosen for its stable role in the industry and for its augmented complexity within an approachable implementation, complexity which could better inform the Zencode implementation.
 
 
 
@@ -64,7 +79,7 @@ The data schemas are added on a per-usecase basis: they refer to specific crypto
 
 Schemas are expressed in a simple format using Lua scripting syntax, for example:
 
-```lua
+```
 -- zencode_keypair
 keypair = S.record {
 	schema = S.Optional(S.string),
@@ -75,13 +90,13 @@ keypair = S.record {
 
 The schema above is the smallest and most commonly used one, composed by one required field and two optional ones, used to validate the input and output of public/private keypairs to be used in transformations.
 
-The only required field in the schema is the `public` key which is validated using the `ECP` type (`S.` is an abbreviation for the `SCHEMA.` namespace). The validation of `S.ECP` is an actual cryptographic validation: Zenroom will check that the big integer number represented by the field corresponds to a valid point on the curve. In case the validation is not passed, the execution of the Zencode script will not take place and Zenroom will return a meaningful error message indicating the wrong field.
+The only required field in the schema is the 'public' key which is validated using the 'ECP' type ('S.' is an abbreviation for the 'SCHEMA.' namespace). The validation of 'S.ECP' is an actual cryptographic validation: Zenroom will check that the big integer number represented by the field corresponds to a valid point on the curve. In case the validation is not passed, the execution of the Zencode script will not take place and Zenroom will return a meaningful error message indicating the wrong field.
 
-The other optional field is the `private` key which can correspond to any sequence of values, therefore no cryptographic validation is possible for it; in this case then the validation used is one that refers to the encoding of the field: `S.hex` is verifying that the value is encoded with a sequence of characters that express only hexadecimal numbers (that is, 0..9 numbers and case-insensitive letters from A to Z). Other encoding tests are also available, for instance `S.base64` if that is the encoding used in the specific implementation.
+The other optional field is the 'private' key which can correspond to any sequence of values, therefore no cryptographic validation is possible for it; in this case then the validation used is one that refers to the encoding of the field: 'S.hex' is verifying that the value is encoded with a sequence of characters that express only hexadecimal numbers (that is, 0..9 numbers and case-insensitive letters from A to Z). Other encoding tests are also available, for instance 'S.base64' if that is the encoding used in the specific implementation.
 
 Another more complex example follows:
 
-```lua
+```
 -- packets encoded with AES GCM
 AES-GCM = S.record {
     checksum = S.hex,
@@ -95,12 +110,12 @@ AES-GCM = S.record {
 }
 ```
 
-In this example no new validations are being used and in fact it just adds fields compared to the previous: it defines a portable packet of ciphertext data that is returned as output of AES-GCM asymmetric encryption as well is accepted as input to AES-GCM decryption. A similarity between these two examples is evident: the presence of the `schema` field. This field is a sort of "introspective" indication matching the data structure to its schema specification. If this field is not present (as it is always optional) then no validation on the data structure will take place, meaning the Zencode implementation leaves the risk (and hopefully the validation task) to the host.
+In this example no new validations are being used and in fact it just adds fields compared to the previous: it defines a portable packet of ciphertext data that is returned as output of AES-GCM asymmetric encryption as well is accepted as input to AES-GCM decryption. A similarity between these two examples is evident: the presence of the 'schema' field. This field is a sort of "introspective" indication matching the data structure to its schema specification. If this field is not present (as it is always optional) then no validation on the data structure will take place, meaning the Zencode implementation leaves the risk (and hopefully the validation task) to the host.
 
-This chapter ends with the current implementation of schema validation data types that are currently implemented for symmetric and asymmetric encryption of ciphertexts as well for implicit certificates. The schema implementation for Zencode is maintained into the sourcecode within the source file `src/lua/zencode_schemas.lua` and can be accessed by the function `ZEN.validate(data,'schema','error')` which is a wrapper of `ZEN.assert(validate(data,schemas['schema']),'error')`.
+This chapter ends with the current implementation of schema validation data types that are currently implemented for symmetric and asymmetric encryption of ciphertexts as well for implicit certificates. The schema implementation for Zencode is maintained into the sourcecode within the source file 'src/lua/zencode_schemas.lua' and can be accessed by the function 'ZEN.validate(data,'schema','error')' which is a wrapper of 'ZEN.assert(validate(data,schemas['schema']),'error')'.
 
 
-```lua
+```
 _G['schemas'] = {
 
    -- packets encoded with AES GCM

views/implicit_certificate.md

@@ -35,7 +35,7 @@ the SEC4 document.
 The second step is the implementation of this formula into the machine
 language executed by the Zenroom VM (a dialect of LUA).
 
-```lua
+```
 -- Zenroom 0.8.0
 -- setup
 random = RNG.new()
@@ -69,7 +69,7 @@ The third step is the improvement of the previous implementation using
 meaningful variable and function names.
 
 
-```lua
+```
 -- Zenroom 0.8.1
 -- setup
 random = RNG.new()
@@ -202,7 +202,7 @@ Scenario 'respond': Alice receives an encrypted message, decrypts it and sends a
 
 The Zencode language is a DSL enforcing a strong declarative behavior underneath and all base data structures are checked against a validation scheme upon input and output. The checks are also of cryptographic nature, for instance public keys are checked to make sure they are actual points on the elliptic curve in use. Here below the data validation schemes so far in use:
 
-```lua
+```
 _G['schemas'] = {
 
    -- packets encoded with AES GCM

views/integration.md

@@ -9,9 +9,9 @@ int zenroom_exec(char *script, char *conf, char *keys,
                  char *data, int verbosity);
 ```
 
-We also have the boilerplate internal to the `script` buffer:
+We also have the boilerplate internal to the 'script' buffer:
 
-```lua
+```
 verbosity_level = 1
 ZEN:begin(verbosity_level)
 ZEN:parse([[
@@ -20,9 +20,15 @@ ZEN:parse([[
 ZEN:run()
 ```
 
-The execution of actual Zencode lines happens sequentially at the time of the `ZEN:run()` call. Each line as part of the whole statement block (utterance) makes use of data types which may or may be validated and should be present in the KEYS and DATA buffers. 
+The execution of actual Zencode lines happens sequentially at the time of the 'ZEN:run()' call. Each line as part of the whole statement block (utterance) makes use of data types which may or may be validated and should be present in the KEYS and DATA buffers. 
+A list of Zenroom/Zencode integrated implementations follow: they have been developed in relation to each pilot software implementation as needed, covering several languages.
 
-A list of Zenroom/Zencode integrated implementations follow: they have been developed in relation to each pilot software implementation as needed, covering several languages. Also notable the presence of the `zenroom` module inside the NodeJS Package Manager collection (NPM) and of course its extremely portable WebAssembly optimized build (universal binary).
+- Go language bindings https://github.com/DECODEproject/zenroom-go
+- Python language bindings https://github.com/DECODEproject/zenroom-py
+- Java (JNI) and SWIG (universal) language bindings are inside Zenroom's source repository https://github.com/DECODEproject/zenroom
+
+Also notable the presence of the 'zenroom' module inside the NodeJS Package Manager collection (NPM) and of course its extremely portable WebAssembly optimized build (universal binary) see https://www.npmjs.com/package/zenroom
+
+Even considering the delay some pilots are presenting especially in having ready the cryptographic contracts that need to be translated to Zenroom and then wrapped into Zencode, it is evident that the way we engineered the Zenroom VM and the Zencode DSL will make it easy to integrate it everywhere without additional work. The main critical concern we hold at this stage of development is the readiness of the cryptographic contracts in which we expect partners to get involved, to push forward the piloting stage.
 
-TODO: list git repos
 

views/introduction.md

@@ -22,5 +22,5 @@ When describing the virtualisation of economic activity in the global context, S
 
 The analysis of legal texts and regulations here shifts into an entirely new domain; it has to refer to conditions that only algorithms can help build or destroy. Thus, referring to this theoretical framework, the research and development of a free and open source language that is intellegible to humans becomes of crucial importance and, from an ethical standing point, DECODE as many other projects in the same space cannot be exempted from addressing it.
 
-When we consider algorithms as contracts regulating relationships (between humans, between humans and nature and, nowadays more increasingly, between different contexts of nature itself) then we should adopt a representation that is close to how the human mind works and that is directly connected to the language adopted. In this thesis I interpret algorithms as the systemic product of complex relationships between contracts and relevant choices made by standing actors [@standing2014Monico]. The ability to verify which algorithms are in place for a certain result to be visualised, to understand and communicate what these algorithms do, to describe and experiment their repercussions on reality is in fact conditioning the very choices standing actors will make.
+When we consider algorithms as contracts regulating relationships (between humans, between humans and nature and, nowadays more increasingly, between different contexts of nature itself) then we should adopt a representation that is close to how the human mind works and that is directly connected to the language adopted. Since algorithms are the systemic product of complex relationships between contracts and relevant choices made by standing actors [@standing2014Monico], the ability to verify which algorithms are in place for a certain result to be visualised becomes very important and should be embedded in every application: to understand and communicate what algorithms and to describe and experiment their repercussions on reality.
 

views/prototypes.md

@@ -8,7 +8,7 @@ At the time of writing our explanation can be based on an extended experimentati
 
 What follows is a brief visualisation of what is realised so far. In particular the first visualisation below refers to the implementation of an asymmetric cryptographic exchange in the fashion of the PGP implementation, based on an exchange of pulic/private keys and their collection into a keyring:
 
-![Asymmetric Diffie-Helman encryption using Zencode](encryption.dot.png)
+![Asymmetric Diffie-Hellman encryption using Zencode](encryption.dot.png)
 
 This simplified flow diagram shows **actual Zencode** that can be executed, higlighting variables that are normally just surrounded by single quotes. Between each code block, which is executed asynchronously as required and at different times, there is a schema which indicates the shape of data in output.
 
@@ -16,5 +16,18 @@ What follows is another flow diagram leading to data outputs that can be reused
 
 ![Implicit certificate issuing and retrieval using Zencode](implicit_certificate.dot.png)
 
+It is very important to understand that the boxes in the flow diagrams shown contain **actual Zencode** meaning that is not just a description, but is source-code that is interpreted and executed by the Zenroom VM to accomplish the tasks described. It is then the main way to faithfully describe what the prototype does internally with the data: each of the prototypes built in DECODE can simply visualize the Zencode that is running to inform any operator of what is going on.
+
+This solution has been realized after trying many different approaches involving visual programming and block programming, which were perhaps richer visually, but less integrated and in general consisting of a way to represent code rather than code itself. The final Zencode solution is also simplier to implement for prototyped host applications.
+
+At the time of writing all functional prototypes in DECODE are embedding Zenroom and can therefore seamlessly implement Zencode without adding more work to implementors, but simply substituting the current Lua based Zenroom scripts to their Zencode implementation. Below a list ofsoftware prototypes also visible at https://github.com/decodeproject
+
+- Mobile app (Zenroom embedded as a react-native javascript component, soon to be converted to native https://github.com/DECODEproject/wallet
+- IoT encoder (Zenroom embedded via Go bindings) https://github.com/DECODEproject/iotencoder
+
+- Chainspace (Zenroom binary executed separately) https://chainspace.io
+
+All DECODE pilots benefit from this development which is successfully integrated through these components. The DECIDIM pilot still needs a working cryptographic implementation of its petition contract in order to be translated to Zencode; the IoT based pilots can all immediately benefit from the Zencode implementation of DH asymmetric encryption based on AES-GCM secure standard; the Amsterdam register pilot can immediately benefit from the Zencode implementation of ECQV implicit certificates.
+
 Future horizons of development of Zencode include further implementations supporting interoperable and extensible crypto schemes on the same EC curve that can still work with the above implementations, as well further refinement of the parser and extension of the schema validation. From this point onwards Zencode must be informed by piloting, while it will be also refined in cooperation with legal experts to match the smart-rule statements so far identified to express consensual data processing conditions.