-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathusage.html
More file actions
283 lines (253 loc) · 25 KB
/
usage.html
File metadata and controls
283 lines (253 loc) · 25 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Usage — xdapy 0.9.0 documentation</title>
<link rel="stylesheet" href="_static/haiku.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="_static/print.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: '',
VERSION: '0.9.0',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>
<script type="text/javascript" src="_static/theme_extras.js"></script>
<link rel="top" title="xdapy 0.9.0 documentation" href="index.html" />
<link rel="next" title="Database Format" href="tables.html" />
<link rel="prev" title="The data model" href="datamodel.html" />
</head>
<body>
<div class="header"><h1 class="heading"><a href="index.html">
<span>xdapy 0.9.0 documentation</span></a></h1>
<h2 class="heading"><span>Usage</span></h2>
</div>
<div class="topnav">
<p>
«  <a href="datamodel.html">The data model</a>
  ::  
<a class="uplink" href="index.html">Contents</a>
  ::  
<a href="tables.html">Database Format</a>  »
</p>
</div>
<div class="content">
<div class="section" id="usage">
<span id="id1"></span><h1>Usage<a class="headerlink" href="#usage" title="Permalink to this headline">¶</a></h1>
<div class="section" id="opening-a-connection">
<h2>Opening a connection<a class="headerlink" href="#opening-a-connection" title="Permalink to this headline">¶</a></h2>
<p>In order to access a database, we need to create a mapper:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">postgresql_mapper</span> <span class="o">=</span> <span class="n">xdapy</span><span class="o">.</span><span class="n">Mapper</span><span class="p">(</span><span class="s">"postgresql://user:pass@host/dbname"</span><span class="p">)</span>
</pre></div>
</div>
<p>we may also connect to a SQLite database:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">sqlite_mapper</span> <span class="o">=</span> <span class="n">xdapy</span><span class="o">.</span><span class="n">Mapper</span><span class="p">(</span><span class="s">"sqlite:///test.db"</span><span class="p">)</span>
</pre></div>
</div>
<p>or even to a memory-based database for testing:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">memory_mapper</span> <span class="o">=</span> <span class="n">xdapy</span><span class="o">.</span><span class="n">Mapper</span><span class="p">(</span><span class="s">"sqlite://"</span><span class="p">)</span>
</pre></div>
</div>
<p>The mapper can also access the profiles in the initialization file defined during the installation.
In this case a connection is explicitly created and passed to the mapper:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">from</span> <span class="nn">xdapy</span> <span class="kn">import</span> <span class="n">Connection</span><span class="p">,</span> <span class="n">Mapper</span>
<span class="n">connection</span> <span class="o">=</span> <span class="n">Connection</span><span class="o">.</span><span class="n">profile</span><span class="p">(</span><span class="s">"demo"</span><span class="p">)</span>
<span class="n">mapper</span> <span class="o">=</span> <span class="n">Mapper</span><span class="p">(</span><span class="n">connection</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="section" id="define-and-create-an-object">
<h2>Define and create an object<a class="headerlink" href="#define-and-create-an-object" title="Permalink to this headline">¶</a></h2>
<p>We now need to define a structure which holds our data. We use the data model from the <a class="reference internal" href="datamodel.html#example2"><em>Science example</em></a> and create the first structure:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="k">class</span> <span class="nc">Experiment</span><span class="p">(</span><span class="n">xdapy</span><span class="o">.</span><span class="n">Entity</span><span class="p">):</span>
<span class="n">declared_params</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">'identifier'</span><span class="p">:</span> <span class="s">'int'</span><span class="p">,</span>
<span class="s">'project'</span><span class="p">:</span> <span class="s">'string'</span><span class="p">,</span>
<span class="s">'author'</span><span class="p">:</span> <span class="s">'author'</span><span class="p">,</span>
<span class="s">'date'</span><span class="p">:</span> <span class="s">'date'</span>
<span class="p">}</span>
</pre></div>
</div>
<p>We have created a subclass of <tt class="xref py py-obj docutils literal"><span class="pre">xdapy.Entity</span></tt> and added a special structure <tt class="docutils literal"><span class="pre">declared_params</span></tt> to it.
In <tt class="docutils literal"><span class="pre">declared_params</span></tt> we have to specify all the parameters we want to store for this <tt class="xref py py-obj docutils literal"><span class="pre">Entity</span></tt>.
Also, we have to specify the type for the respective parameters.
The <tt class="docutils literal"><span class="pre">declared_params</span></tt> structure is evaluated during the creation of the class itself (read: metaclass) and automatically sets some more attributes on <tt class="xref py py-obj docutils literal"><span class="pre">Entity</span></tt>.
This is mainly needed for SQL storage but also provides us with a minimal type checking system.
Without a type checking, the database could save a parameter called <tt class="xref py py-obj docutils literal"><span class="pre">project</span></tt> or <tt class="xref py py-obj docutils literal"><span class="pre">proect</span></tt> and would not complain about a difference.
During a search for the keyword <tt class="xref py py-obj docutils literal"><span class="pre">project</span></tt>, the wrongly spelled parameter could not be found.
Since the parameter declarations are permanently stored during the object registration, an <tt class="xref py py-obj docutils literal"><span class="pre">Exception</span></tt> is raised if a misspelled parameter is stored.
That also means that no other parameter, except for the declared ones, can be stored with an object.</p>
<p>However, Xdapy does not enforce that each parameter from the declared parameters dictionary is defined in every object instance.
If such an enforcement is desired, it needs to be implement in the application.</p>
<p>Every use of Xdapy requires a registration of the objects in the mapper.
With the registration the parameter declarations are stored in the database as triplets containing the class name of the object, parameter name, and parameter type.
The registration process is a security step to ensure compatibility between different applications of Xdapy:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">mapper</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">Experiment</span><span class="p">)</span>
</pre></div>
</div>
<p>If the object class definitions are stored in a file, say objects.py, then they can simply be imported and registered in the mapper.
It is therefore not necessary to rewrite the object definitions with every use.</p>
<p>We can now instantiate and use our <tt class="xref py py-obj docutils literal"><span class="pre">Experiment</span></tt>.
Parameters are either added inside the <tt class="docutils literal"><span class="pre">__init__</span></tt> method or accessed and changed through the <tt class="docutils literal"><span class="pre">params</span></tt> dict:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">experiment</span> <span class="o">=</span> <span class="n">Experiment</span><span class="p">(</span><span class="n">project</span><span class="o">=</span><span class="s">"visual"</span><span class="p">)</span>
<span class="n">experiment</span><span class="o">.</span><span class="n">params</span><span class="p">[</span><span class="s">"identifier"</span><span class="p">]</span> <span class="o">=</span> <span class="mi">5005</span>
<span class="n">experiment</span><span class="o">.</span><span class="n">params</span><span class="p">[</span><span class="s">"author"</span><span class="p">]</span> <span class="o">=</span> <span class="s">"John Jo"</span>
<span class="n">experiment</span><span class="o">.</span><span class="n">params</span><span class="p">[</span><span class="s">"date"</span><span class="p">]</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
</pre></div>
</div>
<p>Using the mapper, we save the object to the database:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">mapper</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="n">experiment</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="section" id="build-an-object-hierarchy">
<h2>Build an object hierarchy<a class="headerlink" href="#build-an-object-hierarchy" title="Permalink to this headline">¶</a></h2>
<p>To show how the hierarchical connections are made, let us first define more object classes and register them:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="c">#Define object classes</span>
<span class="k">class</span> <span class="nc">Session</span><span class="p">(</span><span class="n">xdapy</span><span class="o">.</span><span class="n">Entity</span><span class="p">):</span>
<span class="n">declared_params</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">'count'</span><span class="p">:</span> <span class="s">'integer'</span><span class="p">,</span>
<span class="s">'date'</span><span class="p">:</span> <span class="s">'date'</span><span class="p">,</span>
<span class="s">'experimentalcondition'</span><span class="p">:</span> <span class="s">'string'</span><span class="p">,</span>
<span class="p">}</span>
<span class="k">class</span> <span class="nc">Trial</span><span class="p">(</span><span class="n">xdapy</span><span class="o">.</span><span class="n">Entity</span><span class="p">):</span>
<span class="n">declared_params</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">'count'</span><span class="p">:</span> <span class="s">'integer'</span><span class="p">,</span>
<span class="s">'stimulus'</span><span class="p">:</span> <span class="s">'integer'</span><span class="p">,</span>
<span class="s">'answercorrect'</span><span class="p">:</span> <span class="s">'boolean'</span>
<span class="p">}</span>
<span class="c">#Register</span>
<span class="n">mapper</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">Session</span><span class="p">)</span>
<span class="n">mapper</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">Trial</span><span class="p">)</span>
</pre></div>
</div>
<p>Then, we create instances of the objects:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">s1</span> <span class="o">=</span> <span class="n">Session</span><span class="p">(</span><span class="n">count</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">date</span><span class="o">=</span><span class="n">datetime</span><span class="o">.</span><span class="n">date</span><span class="o">.</span><span class="n">today</span><span class="p">(),</span> <span class="n">experimentalcondition</span><span class="o">=</span><span class="s">'outline'</span><span class="p">)</span>
<span class="n">s2</span> <span class="o">=</span> <span class="n">Session</span><span class="p">(</span><span class="n">count</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">date</span><span class="o">=</span><span class="n">datetime</span><span class="o">.</span><span class="n">date</span><span class="o">.</span><span class="n">today</span><span class="p">(),</span> <span class="n">experimentalcondition</span><span class="o">=</span><span class="s">'filled'</span><span class="p">)</span>
<span class="n">t1</span> <span class="o">=</span> <span class="n">Trial</span><span class="p">(</span><span class="n">count</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">stimulus</span><span class="o">=</span><span class="mi">20</span><span class="p">,</span> <span class="n">answercorrect</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">t2</span> <span class="o">=</span> <span class="n">Trial</span><span class="p">(</span><span class="n">count</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">stimulus</span><span class="o">=</span><span class="mi">36</span><span class="p">,</span> <span class="n">answercorrect</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">t3</span> <span class="o">=</span> <span class="n">Trial</span><span class="p">(</span><span class="n">count</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span> <span class="n">stimulus</span><span class="o">=</span><span class="mi">8</span><span class="p">,</span> <span class="n">answercorrect</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
<span class="n">t4</span> <span class="o">=</span> <span class="n">Trial</span><span class="p">(</span><span class="n">count</span><span class="o">=</span><span class="mi">4</span><span class="p">,</span> <span class="n">stimulus</span><span class="o">=</span><span class="mi">87</span><span class="p">,</span> <span class="n">answercorrect</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
<span class="n">t5</span> <span class="o">=</span> <span class="n">Trial</span><span class="p">(</span><span class="n">count</span><span class="o">=</span><span class="mi">5</span><span class="p">,</span> <span class="n">stimulus</span><span class="o">=</span><span class="mi">26</span><span class="p">,</span> <span class="n">answercorrect</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">t6</span> <span class="o">=</span> <span class="n">Trial</span><span class="p">(</span><span class="n">count</span><span class="o">=</span><span class="mi">6</span><span class="p">,</span> <span class="n">stimulus</span><span class="o">=</span><span class="mi">74</span><span class="p">,</span> <span class="n">answercorrect</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">t7</span> <span class="o">=</span> <span class="n">Trial</span><span class="p">(</span><span class="n">count</span><span class="o">=</span><span class="mi">7</span><span class="p">,</span> <span class="n">stimulus</span><span class="o">=</span><span class="mi">20</span><span class="p">,</span> <span class="n">answercorrect</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
<span class="n">t8</span> <span class="o">=</span> <span class="n">Trial</span><span class="p">(</span><span class="n">count</span><span class="o">=</span><span class="mi">8</span><span class="p">,</span> <span class="n">stimulus</span><span class="o">=</span><span class="mi">16</span><span class="p">,</span> <span class="n">answercorrect</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">t9</span> <span class="o">=</span> <span class="n">Trial</span><span class="p">(</span><span class="n">count</span><span class="o">=</span><span class="mi">9</span><span class="p">,</span> <span class="n">stimulus</span><span class="o">=</span><span class="mi">96</span><span class="p">,</span> <span class="n">answercorrect</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
<span class="n">t10</span> <span class="o">=</span> <span class="n">Trial</span><span class="p">(</span><span class="n">count</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> <span class="n">stimulus</span><span class="o">=</span><span class="mi">36</span><span class="p">,</span> <span class="n">answercorrect</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
</pre></div>
</div>
<p>Finally, the relationships among the objects will be defined through parent and child definitions.
There are several equivalent ways to define relations:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">s2</span><span class="o">.</span><span class="n">parent</span> <span class="o">=</span> <span class="n">experiment</span>
<span class="n">experiment</span><span class="o">.</span><span class="n">children</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">s1</span><span class="p">)</span>
<span class="n">s1</span><span class="o">.</span><span class="n">children</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">t1</span><span class="p">)</span>
<span class="n">s1</span><span class="o">.</span><span class="n">children</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">t2</span><span class="p">)</span>
<span class="n">s2</span><span class="o">.</span><span class="n">children</span> <span class="o">+=</span> <span class="p">[</span><span class="n">t3</span><span class="p">,</span><span class="n">t4</span><span class="p">,</span><span class="n">t5</span><span class="p">,</span><span class="n">t6</span><span class="p">,</span><span class="n">t7</span><span class="p">,</span><span class="n">t8</span><span class="p">,</span><span class="n">t9</span><span class="p">,</span><span class="n">t10</span><span class="p">]</span>
<span class="c">#Save all relations</span>
<span class="n">mapper</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="n">experiment</span><span class="p">)</span>
</pre></div>
</div>
<p>It should be enough to save the highest object in the hierarchy and the relations and children are saved with it.
The connections we just created result in this tree:</p>
<div class="figure">
<img alt="_images/exampleExp.png" src="_images/exampleExp.png" />
</div>
</div>
<div class="section" id="attach-entities">
<h2>Attach entities<a class="headerlink" href="#attach-entities" title="Permalink to this headline">¶</a></h2>
<p>Now the annotations about the observer that participated in the first session are <em>connect</em> or <em>attach</em>:</p>
<div class="highlight-python"><pre> class Observer(xdapy.Entity):
declared_params = {
'name': 'string',
'birthyear': 'integer',
'initials': 'string',
'handedness': 'string',
'glasses': 'boolean'
}
cs = Observer(name="Clara Sight", initials="CS", handedness="right", glasses=False, birthyear=1989)
tt = Observer(name="Tom Token", initials="TT", handedness="right", glasses=True, birthyear=1978)
s1.attach("Observer", cs)
// or, alternatively
s2.context["Observer"].add(tt)</pre>
</div>
<p>Please note, that when attaching an entity to another, a label is provided with it. In this example the label is “Observer”.
This label will be used during searches.</p>
</div>
<div class="section" id="adding-data">
<h2>Adding data<a class="headerlink" href="#adding-data" title="Permalink to this headline">¶</a></h2>
<p>A last critical feature that belongs to the creation and storage of objects is to add data such as:</p>
<ul class="simple">
<li>files</li>
<li>raw data</li>
<li>binary data in general</li>
</ul>
<p>For example you might want to store a file containing the project proposal and its goals with the experiment.
Adding binary data often needs special handling, since its potentially large data sets should not be automatically retrieved and loaded into memory from the database.
Therefore, a special data API is integrated, acting on the <tt class="xref py py-obj docutils literal"><span class="pre">Entity.data</span></tt> property:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">experiment</span><span class="o">.</span><span class="n">data</span><span class="p">[</span><span class="s">"project proposal"</span><span class="p">]</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
<span class="n">experiment</span><span class="o">.</span><span class="n">data</span><span class="p">[</span><span class="s">"dataset #1"</span><span class="p">]</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">more_data</span><span class="p">)</span>
</pre></div>
</div>
<p><em>Xdapy</em> takes care of splitting the data into smaller chunks which do not flood the memory and which are saved to the database right away.
Consequently, the data should not be retrieved and loaded into memory but directly saved to a file:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">save_to</span><span class="p">,</span> <span class="s">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">experiment</span><span class="o">.</span><span class="n">data</span><span class="p">[</span><span class="s">"project proposal"</span><span class="p">]</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="section" id="load-objects-from-the-database">
<h2>Load objects from the database<a class="headerlink" href="#load-objects-from-the-database" title="Permalink to this headline">¶</a></h2>
<p>Assuming that all sessions of two Experiment e1 and e2 were
recorded and stored, the question is how to extract the data
from the database for analysis. There are several possibilities
to query the database. The simplest options are the mapper
methods find_all and find_first:</p>
<div class="highlight-python"><pre>o = m.find_first(Observer, filter={"birthyear": range(1970, 1985)})
print o.params
o = m.find_all(Observer, filter={"initials": ["%S%"]})
print o[0].params
>>>>{u'birthyear': 1978, u'initials': u'TT', u'handedness': u'right', u'name': u'Tom Token', u'glasses': True}
>>>>{u'birthyear': 1989, u'initials': u'CS', u'handedness': u'right', u'name': u'Clara Sight', u'glasses': False}</pre>
</div>
</div>
<div class="section" id="query-the-database">
<h2>Query the database<a class="headerlink" href="#query-the-database" title="Permalink to this headline">¶</a></h2>
<p>Admittedly, the syntax of complex searches is not easy to
read. We therefore recommend to encapsulate repeatedly used searches in regular Python methods</p>
<p>The following procedure lists the stimulus number and response of the trials from the outline condition in the visual experiment
in which observer CS responded correctly.</p>
<div class="highlight-python"><pre>sessions = mapper.find_complex("Session",
{("_context", "Observer"): {"_any": [("Observer",{"initials": "CS"})]},
"_parent":("Experiment",{"project":"vision"}),
"experimentalcondition":"outline"})
trials = m.find_complex("Trial",{"_parent": {"_any": sessions}})
for trial in trials:
print trial.params["stimulus"], trial.params["answercorrect"]
>>>> 20 True
>>>> 36 True</pre>
</div>
</div>
</div>
</div>
<div class="bottomnav">
<p>
«  <a href="datamodel.html">The data model</a>
  ::  
<a class="uplink" href="index.html">Contents</a>
  ::  
<a href="tables.html">Database Format</a>  »
</p>
</div>
<div class="footer">
© Copyright 2012, Hannah Dold, Rike-Benjamin Schuppner.
Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> 1.1.3.
</div>
</body>
</html>