March 28, 2010

Loading CSV data in Google Apps Engine datastore

Edit your app.yaml, and add the following lines to the handlers: section:

- url: /remote_api
  script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
  login: admin

1.Create hist.py similar to the type of data in CSV file.

   1: from google.appengine.ext import db

2:

   3: class HistoricalPrices(db.Model):

   4:     SYMBOL = db.StringProperty()

   5:     SERIES  = db.StringProperty()

   6:     OPEN = db.FloatProperty()

   7:     HIGH = db.FloatProperty()

   8:     LOW = db.FloatProperty()

   9:     CLOSE = db.FloatProperty()

  10:     LAST = db.FloatProperty()

  11:     PREVCLOSE = db.FloatProperty()

  12:     TOTTRDQTY = db.FloatProperty()

  13:     TOTTRDVAL = db.FloatProperty()

  14:     TIMESTAMP = db.DateProperty()

  15:     DUMMY1 = db.StringProperty()

16:

2.Now Create the loader code StockLoader.py

   1: import datetime

   2: from google.appengine.ext import db

   3: from google.appengine.tools import bulkloader

   4: import hist

5:

   6: class HistoryLoader(bulkloader.Loader):

   7:     def __init__(self):

   8:         bulkloader.Loader.__init__(self, 'HistoricalPrices',

   9:                                    [('SYMBOL', str),

  10:                                     ('SERIES', str),

  11:                                     ('OPEN', float),

  12:                                     ('HIGH', float),

  13:                                     ('LOW', float),

  14:                                     ('CLOSE', float),

  15:                                     ('LAST', float),

  16:                                     ('PREVCLOSE', float),

  17:                                     ('TOTTRDQTY', float),

  18:                                     ('TOTTRDVAL', float),

  19:                                     ('TIMESTAMP',

  20:                                      lambda x: datetime.datetime.strptime(x, '%d-%b-%Y').date()),

  21:                                     ('DUMMY1', str)

  22:                                    ])

23:

  24: loaders = [HistoryLoader]

3.Following commands to execute the above.

"D:Program FilesGooglegoogle_appengineappcfg.py" upload_data --config_file=StockLoader.py --filename=test.csv --kind=HistoricalPrices "D:devappsengineproxy"

where test.csv is something like following

UNITECH,EQ,82.9,83.8,82.45,82.8,82.8,82.3,19904383,1652618271.55,4-JAN-2010,

UNITECH,EQ,83.5,84.9,83.4,84.1,84,82.8,34197626,2882001214.7,5-JAN-2010,

UNITECH,EQ,84.5,85.45,83.2,84.85,84.7,84.1,34704003,2936028889.35,6-JAN-2010,

UNITECH,EQ,85.2,85.7,83.85,84.15,84.15,84.85,28855332,2446134892.4,7-JAN-2010,

last directory mentioned is the directory of the apps engine project directory where app.yaml file exists

ref http://code.google.com/appengine/docs/python/tools/uploadingdata.html

Posts by Year

Loading CSV data in Google Apps Engine datastore