Causality Causality Workbench                                                             Challenges in Machine Learning Causality

What is the Virtual Laboratory?

The Virtual Laboratory allows researchers on causality to perform experiments on artificial systems in order to unravel causal relationships. Download our Technical Report.

One important goal of causal modeling is to unravel enough of the data generating process to be able to predict the consequences of actions (also called interventions, experiments or manipulations), performed by an external agent. This setup violates the classical i.i.d. assumption commonly made in machine learning. For instance, in policy-making, one may want to predict "the effect on a population health status" of "forbidding to smoke in public places", before passing a law.

While many algorithms have been proposed recently to learn causal relationships from non-experimental data (simple observations without external intervention), experimentation is still considered the only ultimate means of validation of a causal model. But, experiments are often costly and sometimes impossible or unethical to perform.

The virtual lab offers the possibility of researchers to experiment with simulated systems (for a list, see the Index page), by setting the values of certain variables and observing others. It features a realistic setup in which the artificial systems model real applications in medicine, marketing, life sciences, social sciences, etc. and the experiments cost a certain amount of virtual cash; simple observations without intervention generally costing less. The predictive power of the model can then be evaluated by providing test data corresponding to new interventions.

To understand the basic concept of experimentation in causal modeling, you may want to first look at these historical examples in epidemiology brought to us by ThinkQuest:

Lung cancer
Smallpox
Food poisoning

How to Design Experiments?

Here is a brief outline of the steps taken in experimenting and causal modeling:
  1. Problem specification: Define your problem and your goals . In the Virtual Lab, problems are already formalized.
  2. Feature set definition: Identify potentially relevant factors . In the virtual lab the feature set is already given: those are the system variables. In some cases, the task designer may hide a number of variables to test the robustness of algorithms against hidden confounders, which are unknown common causes to several variables in your system.
  3. Manipulation protocol: Figure out how to perform actions on the system and manipulate variables of interest. This step is often very complex in real experiments because we do not always have easy means of influencing variables individually as an external agent. Not all variables are actionable or even observable. Some may be unethical to manipulate. In the Virtual Lab, things are simple: we tell you which variables are actionable. All you have to do to carry out experiments is to initialize or clamp desired variables.
  4. Experimental design: Given a budget (here you have "virtual cash"), spend it in data collection, observations, and manipulations to achieve the goals you have set to yourselves.
  5. Modeling: Carry out the experiments and build models with the data collected. Eventually iterate this process until a satisfactory model is obtained. In the Virtual Lab, all you have to do is to submit queries via the Upload Page using the format described below. Your virtual cash account will be automatically debited and you will be able to download the results of your experiments your private Mylab page.
  6. Deployment: Deploy your model to predict the consequences of actions in new situations. In the Virtual Lab, we provide you with test data, which was drawn from a post-manipulation distribution. The manipulations are performed by the task designers. depending on the task, the designers may or may not inform you of what exact manipulation(s) was performed in test data. When you are done with modeling and before your run out of virtual cash, you must ask for the test data. WARNING: The test data will cost you virtual cash, so make sure you keep enough virtual cash. We do not withhold from your cash account a fixed amount to pay for the test data because, if you cleverly design your experiments and your model, you might get it at a discount price by querying only a subset of the variables. Once you ask for test data, you must return your prediction results on test data, no query for more data are allowed.
We will organize competitions in the future. In a competition setup, it will not be possible to work several times on the same task. However, for the time being, you are free to experiment multiple times on the same problem and even to run concurrent experiments with different strategies.

How to Submit Queries and Get Data?

Easy start

The submission of data requests and prediction results is via the Upload page. A submitted query should be a zip file bundling files described below. Use
zip query.zip *
or
tar cvf query.tar *; gzip query.tar
to create valid archives. We provide several examples of queries for the LUCAS model:
  1. Observations. Request 25 examples of all the variables. No manipulation is performed. observational data only.
  2. Experiment 1.. Request 10 values of the target variable. Most covariate values are provided, except a few missing values.
  3. Experiment 2.. Not all variables are manipulated. The pre-manipulation values are given by the selection of training samples.
  4. Test data. We ask for the test set 2. Here we ask for postmanip variables, but we will not get them because the test data does not include any postmanipulation observations.
  5. Default training set. Training data can be purchased unlabeled, this is cheaper. Then the labels may be queried separately.
  6. Survey data. Query asking for a subset of the labels of the default training set.
  7. Prediction results. Predictions of the target post-manipulation values on test set 2.
If you want to get baseline results without experimenting, it is always possible with the initial budget to buy the default training set and the entire test set. Just submit two (separate) queries with a single query file, each containing a single word:
  1. To get training data, write the word TRAIN on the first line.
  2. To get test data, write the word TEST on the first line.

File formats for data queries and prediction results

Filename Non-experimental data Experimental data Survey data Prediction results Description File Format
[submission].query
Compulsory (TRAIN, TEST [n] or OBS [num])
Compulsory
(EXP)
Optional
(SURVEY)
Optional
(PREDICT [n])
Type of query.
A single key word on the first line, optionally followed by a number on the same line:
  TRAIN: get the default training set.
  TEST [n]; replace [n] by 1, 2, 3 to get the nth test set TEST=TEST 1.
  OBS [num]: get observational data; replace [num] by the number of samples requested.
  EXP: get experimental data (the number of samples is determined by the number of lines in [submission].sample and [submission].manipval).
  SURVEY: get training labels.
  PREDICT [n]: replace [n] by 1, 2, 3 to indicate that predictions correspond to the nth test set. PREDICT=PREDICT 1.
[submission].sample
NA
Optional
Compulsory
NA
Sample ID in the default training set. The corresponding samples are used to set premanipulation values.
A list of sample numbers, one per line (the numbering is 1-based and corresponds to lines in the training data).
[submission].premanipvar
Optional
Optional
Optional
NA
List of the pre-manipulation variables (observed before or without experimentation). By default (no file given): (1) for non-experimental and experimental data: all the observable variables, except the target; (2) for survey data: the target.
A space-delimited  list of variable numbers on the first line of the file. All variables are numbered from 1 to the maximum number of visible variables, except the target variable (if any), which is numbered 0.
[submission].manipvar
NA Compulsory
NA
NA
List of the variables to be manipulated (clamped).
[submission].postmanipvar
NA
Compulsory
NA
Optional
List of the post-manipulation variables  (observed after experimentation). By default: the target variable.
[submission].premanipval
NA
NA
NA
NA
Not applicable: use [submission].sample to initialize values of the pre-manipulation variables.
Each line corresponds to an instance (sample) and should contain space delimited variable values for all the variables of that instance. Use NaN if the value is missing or omitted.
The number of lines in [submission].manipval should match the number of samples in [submission].sample (if provided).
You may omit [submission].query and provide
[submission].predict instead of [submission].postmanipval if there is a single test set and no experiments are involved.
[submission].manipval
NA
Compulsory
NA
NA
Clamped values for the manipulated variables, listed in [submission].manipvar.
[submission].postmanipval
or
[submission].predict
NA
NA
NA
Compulsory
Predictions values for all the samples of TESTn.

File formats for data received and prediction scores

Data archives with the training or test data you requested are available from your private Mylab a short time after you placed your query. Prediction score are also displayed on the Leaderboard page.

Filename Non-experimental training data Survey data Experimental training data Test data Evaluation score Description File Format
[submission].query
Optional
(TRAIN or OBS)
Optional
SURVEY
Optional
EXP
Optional
TEST
Optional
PREDICT
Type of query.
One keyword and optionally a number on the first line (copied from the query submitted).
[answer].premanipvar
Optional
NA
Present if requested
Present
NA
List of the pre-manipulation variables. By default, all the observable variables except the target.
A space-delimited  list of variable numbers on the first line of the file. All variables are numbered from 1 to the maximum number of visible variables, except the default target variable (if any), which is numbered 0. If the file is missing or empty, an empty list is assumed.
[answer].manipvar
NA NA
Present
Optional
NA
List of the variables to be manipulated (clamped).  
 [answer].postmanipvar
NA
NA
Optional
Optional
NA
List of the post-manipulation variables. By default: the target variable 0.
[answer].premanipval or
[answer].data
Present
NA
Present if requested
Present
NA
Pre-manipulation values.
Each line corresponds to an instance (sample) and contains space delimited variable values for all the variables of that instance (or a single target value for [answer].label files). [answer].data files contain unlabeled default training data for problems without experimentation
[answer].label
NA (to get the target variable values, use the index 0)
Present
NA (to get the target variable values, use the index 0)
NA
NA
Target values for default trainign examples. Equivalent to [answer].premanipval when [answer].premanipvar (with the single value 0) is omitted.
[answer].manipval
NA
NA
Present
Optional
NA
Clamped values for the manipulated variables, listed in [answer].manipvar. Those  correspond to manipulations performed by the organizers so they are free of charge.
[answer].postmanipval
NA
NA
Present
Hidden to the participants
NA
Post-manipulation values. In answer to [submission].postmanipvar
[answer].is_overbudget
Optional
Optional
Optional
Optional
Optional
File indicating that the budget was overspent and the query was not processed.
The value1.
[answer].score
NA
NA
NA
NA
Present
Prediction score.
A numeric value.
[answer].ebar
NA
NA
NA
NA
Present
Error bar.
A numeric value.
[answer].varnum
Present
Present
Present
Present
NA
The total number of observable variables (excluding the target).
A numeric value.
[answer].samplenum
Present
Present
Present
Present
NA
Number of samples requested.
A numeric value.
[answer].obsernum
Present
Present
Present
Present
NA
Number of variable values observed (including the target).
A numeric value.
[answer].manipnum
Present
Present
Present
Present
NA
Number of values manipulated.
A numeric value.
[answer].targetnum
Present
Present
Present
Present
NA
Number of target values observed.
A numeric value.
[answer].samplecost
Present
Present
Present
Present
NA
Cost for the samples requested (labeled samples may cost more than unlabeled samples).
A numeric value.
[answer].obsercost
Present
Present
Present
Present
NA
Cost for the observations made.
A numeric value.
[answer].manipcost
Present
Present
Present
Present
NA
Cost for the manipulations made.
A numeric value.
[answer].targetcost
Present
Present
Present
Present
NA
Additional cost for target observations.
A numeric value.
[answer].totalcost
Present
Present
Present
Present
NA
Total cost.
A numeric value.