Paypal background

Paypal offers different levels of integration, which, depending on what you need to do, might be better suited for your needs. It is important that you get to know at least the basic integration concepts that paypal provides before starting to program anything so that you plan in advance what is best suited to your needs.

That said, let me try to give you a rough idea of the different levels involved before going any further so as to better understand the little area that this article covers. It is, however, an area in which most of the small and middle sized projects may fall into.

In broad lines, there are 3 levels of integration that one may achieve with Paypal:

• Express Checkout: Within a seller account in paypal, you can create buttons with information related to each item that you may be selling, (name, description, item number, and pricing). You can have up to 1.000 different buttons or items, defined in this way. After that, it is a matter of setting the buttons on the html to go along with the application. Regarding web2py, it is really simple to just copy the code that paypal creates for each button in a text field in your product db, and then just present it on the screen whenever its needed.

  Using this method, one can opt for different purchase experiences including straight checkout or cart management (managed by paypal) which would let you add or remove items from within the checkout screen in paypal.

  I do not like this method, unless you would be selling very very few item codes, as it may get to be a pain to maintain your articles in paypal. If you are selling a few services or whatever with a small set of prices, it might very well be worth it, as you don't have to work much from the programming point of view and its really simple to set up.

• Standard Integration: This is the one that we will be covering in this article. It basically lets you manage your own product database etc, and send all the data to paypal, at the moment of payment, so that the whole checkout process is managed at paypal. After the transaction has been completed, you can choose (as per configuration of your profile in your paypal seller account) wether the customer is redirected back to your domain (you can setup a default URL to return to, or send that URL dinamically each time you send the data for the checkout, but the functionality needs to be activated in your seller account).

  Two things need to be mentioned here, which I feel are part of the Standard Integration, although they are not required in order to have your basic site working:

  • PDT: Payment Data Transfer, which would be the process by which the customer is sent back to your domain, which lets you capture the transaction data (payment confirmation data from paypal), and show it in a confirmation screen in your own domain, with any further information you may want to show, or redirect him to continue his shopping. It is not completelly safe, as nothing garanties that the customer will be redirected, this may well happen, because on some cases, paypal doesn't execute the redirection, but forces the customer to click on an extra button to return to your domain, so as to give the opportunity to the customer to join paypal. This happens whenever the customer pays by credit card and not using his paypal account.

  • IPN: Instant Payment Notification, which is a messaging service that connects to your domain to send the information of each transaction processed at Paypal. It doesn't stop sending the message until you acknowledge its reception (or 4 days pass without acknowledgement). This is the safest way to collect all the data from all the transactions processed at paypal, and trigger any internal process that you may have, ussually you will want to do the shipping of your products at this point.

• Detailed Integration: In here I am really grouping a number of other methods ands APIs, that I will not be detailing, some of them for very specific uses. The only that that I would like to mention more specifically is NVP (Name Value Pairs), as I feel gives you a very simple programing interface with wich you can do very detailed processes controlling all your data, and all your transaction flow from your domain. Using NVP, you can for example, capture all the data related to a payment in your domain, and only at that point, send all the information to paypal to process the payment (as opposed to processing the checkout which is what we are doing in the previous items). You have a good example as to how to implement this at http://mdp.cti.depaul.edu/appliances/default/show/28 or go to main webpage <www.web2py.com> and find it under free applications, PaypalEngine developed by Matt Sellers. You should however check the detailed documentation at paypal as the process involves many steps in order to ensure the maximum security of your transactions.

So basically, in Express Checkout paypal manages your cart (and master data), the checkout process and of course, payments. With Standard Integration paypal manages checkout and payments, and with further detailed integration, you can make it so that it manages only the payments.

First Steps

Before moving on, all the technical documentation regarding integration with paypal, can be found at:

https://cms.paypal.com/us/cgi-bin/?cmd=_render-content&content_ID=developer/library_documentation.

A link to this URL in case this changes can be found by clicking on the Documentation link at https://developer.paypal.com/.

So moving on to how to use the standard integration, the first thing you should do, is create yourself a sandbox account. You do this at https://developer.paypal.com/ create yourself an account, and once logged in, create at least two test accounts, seller and a buyer respectivelly. There is a good guide on all the necessary steps called PP sandbox user guide which you can find at the documentation link provided before, or on an html version at https://cms.paypal.com/us/cgi-bin/?cmd=_render-content&content_ID=developer/howto_testing_sandbox. Everything on how to set your account up and start running is described there.

Once you have that setup and running, you will have your seller ID and email (you can use any of them to identify yourself to paypal on the code below, although I prefer the ID, if only to avoid possible spam).

Checkout

Ok, so now, we can already create the checkout button that will take our customers to the paypal site with all our cart data. Before moving further, you can find all documentation related to this point at the documentation link provided before under Website Payments Standard Integration Guide or directly in html format at:

https://cms.paypal.com/us/cgi-bin/?cmd=_render-content&content_ID=developer/howto_html_wp_standard_overview

Namelly check the information about Third-Party Shopping Carts. Anyway, creating the button to send all the information is actually very simple, all that is needed is the following code in your checkout page view:

\begin{lstlisting}
<form action="https://www.sandbox.paypal.com/cgi-bin/webscr" method="post">
  <!-- Select the correct button depending on country etc. 
  If you can do it with pregenerated buttons (with prices included etc) 
  then so much the better for security -->
  <input type="hidden" name="business" value="{{=paypal_id}}" />
<input type="image" src="https://www.sandbox.paypal.com/es_XC/i/btn/btn_buynowCC_LG.gif" border="0" name="submit" alt="PayPal - The safer, easier way to pay online!">
<img alt="" border="0" src="https://www.sandbox.paypal.com/es_XC/i/scr/pixel.gif" width="1" height="1">
<form action="http://www.sandbox.paypal.com/cgi-bin/webscr" method="post" />
<input type="hidden" name="cmd" value="_cart" /> 
<input type="hidden" name="upload" value="1" /> 
<input type="hidden" name="charset" value="utf-8">
<input type="hidden" name="currency_code" value="EUR" />
<input type="hidden" name="display" value="1"/>
<input type="hidden" name="shopping_url" value="http://www.micropolixshop.com/giftlist/default/glist"/> <!-- Not really necessary, only if want to allow continue Shopping -->
<input type="hidden" name="notify_url" value="http://www.micropolixshop.com/giftlist/default/ipn_handler"/> <!-- Or leave blank and setup default url at paypal -->
<input type="hidden" name="return" value="http://www.micropolixshop.com/giftlist/default/confirm"/> <!-- Or leave blank and setup default url at paypal -->
<input type="hidden" name="custom" value="{{=session.event_code}}"/>
{{k=1}}
{{for id,product in products.items():}}
  <input type="hidden" name="item_number_{{=k}}" value="{{=product.ext_code}}"/> 
  <input type="hidden" name="item_name_{{=k}}" value="{{=product.name}}"/> 
  <input type="hidden" name="quantity_{{=k}}" value="{{=session.cart[str(id)]}}"/>
  <input type="hidden" name="discount_rate_{{=k}}" value="15"/> <!-- ie, wants a 15% on all articles always -->
  <input type="hidden" name="tax_{{=k}}" value="{{=product.price*product.tax_rate}}"/>
  <input type="hidden" name="amount_{{=k}}" value="{{=product.price}}"/>
  {{k+=1}}
{{pass}}
</form>
\end{lstlisting}

A couple of comments regarding Listing lst:CheckoutButton:

• In all cases, to move from sandbox to production, the url to use only needs to change from https://www.sandbox.paypal.com to https://www.paypal.com

• You can create the buttons using the create new button functionality at your seller account, and then reuse the code it would give you having chosen language and the type of button to use. That way, you will get the correct link to the image to be used for your paypal button.

• The field cmd with value _cart is very important, read the documentation to see the possible values of this field depending on what you want to do. I am assuming a cart scenario on this example.

• The fields shopping_url, notify_ulr and return, can be omited if you setup your seller account profile. If you set it up here, this takes precedence over the default values setup in your seller account.

• The field custom I think is rather important, as is one of the few fields that let you introduce data not shown to the customer, that may allow you to track any extra information. It is per transaction (not per item). In this case, I choose to use an internal event code to track all purchases related to an event (special promotion if you like or whatever).

• As you can see, I create a loop with all the cart items to do the checkout by passing a dictionary with all the product data. I have the information of the items purchased in the session. They get named and numbered following the paypal rules.

• Regarding the discount, even though you set the discounts per item, paypal, only shows a discount total, I do not know if this is different in the Pro version.

For more information, you should check the documentation named before, which includes a list of all the available fields to you (which include shipping charges etc).

Checkout Confirmation / Payment Data Transfer

Once the customer finishes paying through paypal, he will be redirected to your website automatically if it is setup in the account and he is already a paypal user (else he will have to click on an button to return to your site). This section shows you how to set your application so that it will receive the payment data confirmation from paypal and show a confirmation to your customer.

You can read detailed documentation on this subject here:

https://cms.paypal.com/us/cgi-bin/?cmd=_render-content&content_ID=developer/howto_html_paymentdatatransfer

where you can see how to set it up in detail, so that you know where to get your token from, which you need to identify yourself to paypal to confirm and get the data. In an case, refer to Figure fig:pdt (picture taken from paypal docs) so as to give you a detailed view of the process flow.

In Listing lst:generic-def I include a number of generic functions that I have used in setting up the interface. The Connection class definition is a modified version of a generic connection example I found surfing the web, but I cannot really recall where. The add_to_cart, remove_from_cart, empty_cart and checkout I include as an example on how to setup your cart, which are taken from EStore which you can find at http://www.web2py.com/appliances/default/show/24 created by Massimo di Pierro.

Again, please understand that I am over simplifying the different
methods to try to explain in a few lines the different
possibilities

\lstset{basicstyle=\footnotesize, frame=shadowbox, rulesepcolor=\color{blue}, breaklines=true, language=Python, caption=Generic Classes and Function definitions, label=lst:generic-def}

# db.py file
#########################################################################
## Global Variables definition
#########################################################################
domain='www.sandbox.paypal.com'
protocol='https://'
user=None
passwd=None
realm=None
headers = {'Content-Type':'application/x-www-form-urlencoded'}
# This token should also be set in a table so that the seller can set it up
# dinamically and not through the code. Same goes for the PAGINATE.
paypal_token="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
PAGINATE = 20
#########################################################################

# default.py file
#########################################################################
# coding: utf8

import datetime
import string

if not session.cart: session.cart, session.balance={},0
app=request.application

# Setup paypal login email (seller id) in the session
# I store paypal_id in a table
session.paypal_id=myorg.paypal_id
import urllib2, urllib
import datetime

class Connection:
  def __init__(self, base_url, username, password, realm = None, header = {}):
    self.base_url = base_url
    self.username = username
    self.password = password
    self.realm    = realm
    self.header   = header

  def request(self, resource, data = None, args = None):
    path = resource

    if args:
      path += "?" + (args)

    # create a password manager
    password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()

    if self.username and self.password:
      # Add the username and password.
      password_mgr.add_password(self.realm, self.base_url, self.username, self.password)

    handler = urllib2.HTTPBasicAuthHandler(password_mgr)

    # create "opener" (OpenerDirector instance)
    opener = urllib2.build_opener(handler)

    # Install the opener.
    # Now all calls to urllib2.urlopen use our opener.
    urllib2.install_opener(opener)
    #Create a Request
    req=urllib2.Request(self.base_url + path, data, self.header)
    # use the opener to fetch a URL
    error = ''
    try:
      ret=opener.open(req)
    except urllib2.HTTPError, e:
      ret = e
      error = 'urllib2.HTTPError'
    except urllib2.URLError, e:
      ret = e
      error = 'urllib2.URLError'

    return ret, error

def add_to_cart():
  """
  Add data into the session.cart dictionary
  Session.cart is a dictionary with id product_id and value = quantity
  Session.balance is a value with the total of the transacion.
  After updating values, redirect to checkout
  """
  pid=request.args[0]
  product=db(db.product.id==pid).select()[0]
  product.update_record(clicked=product.clicked+1)
  try: qty=session.cart[pid]+1
  except: qty=1
  session.cart[pid]=qty
  session.balance+=product.price
  redirect(URL(r=request,f='checkout'))

def remove_from_cart():
  """
  allow add to cart
  """
  pid=request.args[0]
  product=db(db.product.id==pid).select()[0]
  if session.cart.has_key(pid):
      session.balance-=product.price
      session.cart[pid]-=1
      if not session.cart[pid]: del session.cart[pid]
  redirect(URL(r=request,f='checkout'))

def empty_cart():
  """
  allow add to cart
  """
  session.cart, session.balance={},0
  redirect(URL(r=request,f='checkout'))

def checkout():
  """
  Checkout
  """
  pids=session.cart.keys()
  cart={}
  products={}
  for pid in pids: 
    products[pid]=db(db.product.id==pid).select()[0]
  return dict(products=products,paypal_id=session.paypal_id)

Finally, Confirm, at Listing lst:confirm will process the information sent from paypal, with the four step process described in Figure fig:pdt steps 2,3,4 and 5.

def confirm():
  """
  This is set so as to capture the transaction data from paypal
  It captures the transaction ID from the HTTP GET that paypal sends.
  And using the token from vendor profile PDT, it does a form post.
  The data from the http get comes as vars Name Value Pairs.
  """
  if request.vars.has_key('tx'):
    trans = request.vars.get('tx')
    # Establish connection.
    conn = Connection(base_url=protocol+domain, username=user, password = passwd, realm = realm, header = headers) 
    data = "cmd=_notify-synch&tx="+trans+"&at="+paypal_token 
    resp,error=conn.request('/cgi-bin/webscr', data)
    data={}
    if error=='':
      respu = resp.read()
      respuesta = respu.splitlines()
      data['status']=respuesta[0]
      if respuesta[0]=='SUCCESS':
        for r in respuesta[1:]:
          key,val = r.split('=')
          data[key]=val
        msg=''
        if data.has_key('memo'): msg=data['memo']
        form = FORM("Quiere dejar un mensaje con los regalos?",
            INPUT(_name=T('message'),_type="text",_value=msg),
            INPUT(_type="submit"))
        if form.accepts(request.vars,session):
          email=data['payer_email'].replace('%40','@')
          id = db.gift_msg.insert(buyer=data['payer_email'],transact=trans,msg=form.vars.message) 
          response.flash=T('Your message will be passed on to the recipient')
          redirect(URL(r=request,f='index'))

        return dict(data=data,form=form)
      return dict(data=data)
    else:
      data['status']='FAIL'
  else:
    redirect(URL(r=request,f='index'))
  return dict(trans=trans)

Just for the shake of completion I am adding a very basic example of confirm.html which you can see in Listing lst:confirmhtml.

{{extend 'layout.html'}}

{{if data['status'] == 'SUCCESS':}}
<p><h3>{{=T('Your order has been received.')}}</h3></p>
<hr>
<b>{{=T('Details')}}</b><br>
<li>{{=T('Name:')}} {{=data['first_name']}} {{=data['last_name']}}</li>
<li>{{=T('Purchases for event:')}}: {{=data['transaction_subject']}}</li>
<li>{{=T('Amount')}}: {{=data['mc_currency']}} {{=data['mc_gross']}}</li>
<hr>
{{=form}}
{{else:}}
{{=T('No confirmation received from paypal. This can be due to a number of reasons, please check your email to see if the transaction was successful.')}}
{{pass}}

{{=T('Your transaction has finished, you should receive an email of your purchase.')}}<br> 
{{=T('In case you have an account at paypal, you can check your transaction details at')}} <a href='https://www.paypal.es'>www.paypal.es</a>

IPN: Instant Payment Notification

As mentioned before, one cannot trust the PDT process to receive the information from all transactions as a great number of things can happen. Thus, you need to implement an additional process if you need to do additional processing of the information from your sales, or if you want to keep a local database of the actual sales processed.

This is done with IPN. You can find all the documentation related at the documentation site URL given previously. You will need to turn on the IPN functionality at your seller account, as well as give a default URL to receive those messages which should be equal to the view in which you process them. In the case of this example it would be: http://www.yourdomain.com/yourapp/default/ipn_handler

The process is quite similar to that of PDT, even the variables are the same. The main difference is that IPN are sent from paypal until you acknowledge them. The view for this function (default/ipn_handler.html) can very well be left blank. I am including also the table definition for logging the messages from paypal.

Anyway, find in Listing lst:ipnhandler is an example of how to set them up

# At models/db.py
######################################################################
db.define_table('ipn_msgs',
  Field('trans_id',label=T('transaction id')),
  Field('timestamp','datetime',label=T('timestamp')),
  Field('type',label=T('type')),
  Field('msg','text',label=T('message')),
  Field('processed','boolean',label=T('processed')),
  Field('total','double',label=T('total')),
  Field('fee','double',label=T('fee')),
  Field('currency',length=3,label=T('currency')),
  Field('security_msg',label=T('security message'))
  )

# At controllers/default.py
######################################################################
def ipn_handler():
  """
  Manages the ipn connection with Paypal
  Ask PayPal to confirm this payment, return status and detail strings
  """
  parameters = None 
  parameters =  request.vars
  if parameters:
    parameters['cmd'] = '_notify-validate'
    params = urllib.urlencode(parameters)
    conn = Connection(base_url=protocol+domain, username=user, password = passwd, realm = realm, header = headers)
    resp,error =conn.request('/cgi-bin/webscr', params)
    timestamp=datetime.datetime.now()
    # We are going to log all messages confirmed by paypal.
    if error =='':
      ipn_msg_id = db.ipn_msgs.insert(trans_id=parameters['txn_id'],timestamp=timestamp,type=resp.read(),msg=params,
        total=parameters['mc_gross'],fee=parameters['mc_fee'],currency=parameters['mc_currency'])
      # But only interested in processing messages that have payment status completed and are VERIFIED by paypal.
      if parameters['payment_status']=='Completed':
        process_ipn(ipn_msg_id,parameters)
\end{lstlisting}

Only thing missing would be to process the information received and
check for errors or possible fraud attempts. You can see an example
function in Listing lst:processipn. Although this is probably
something that would change quite a bit from one project to the
next, I hope that it may serve you as a rough guide.
\lstset{basicstyle=\footnotesize, frame=shadowbox, rulesepcolor=\color{blue}, breaklines=true, language=Python, caption=IPN message processing, label=lst:processipn}

\begin{lstlisting}
def process_ipn(ipn_msg_id,param):
  """
  We process the parameters sent from IPN paypal, to correctly store the confirmed sales
  in the database.

  param -- request.vars from IPN message from paypal
  """
  # Check if transaction_id has already been processed.
  query1 = db.ipn_msgs.trans_id==param['txn_id']
  query2 = db.ipn_msgs.processed == True
  rows = db(query1 & query2).select()
  if not rows:
    trans = param['txn_id']
    payer_email = param['payer_email']
    n_items = int(param['num_cart_items'])
    pay_date = param['payment_date']
    total = param['mc_gross']
    curr = param['mc_currency']
    event_code = param['custom']
    if param.has_key('memo'): memo=param['memo']
    event_id = db(db.event.code==event_code).select(db.event.id)
    if not event_id:
      db.ipn_msgs[ipn_msg_id]=dict(security_msg=T('Event does not exist'))
    else:
      error=False
      for i in range(1,n_items+1):
        product_code = param['item_number'+str(i)]
        qtty = param['quantity'+str(i)]
        line_total = float(param['mc_gross_'+str(i)]) + float(param['mc_tax'+str(i)])
        product=db(db.product.ext_code==product_code).select(db.product.id)
        if not product:
          db.ipn_msgs[ipn_msg_id]=dict(security_msg=T('Product code does not exist'))
          error=True
        else:
          db.glist.insert(event=event_id[0],product=product[0],buyer=payer_email,transact=trans,
              purchase_date=pay_date,quantity_sold=qtty,price=line_total,observations=memo)
      if not error: db.ipn_msgs[ipn_msg_id]=dict(processed=True)

Closing up

I hope that this guide has helped you to set up your paypal site using web2py, or at least helped you understand the basic concepts behind setting up one and the different possibilities that you have available.

Benigno Calvo Adiego, author of the present article, is co-founder of \ad http://www.albendas.com and executive director of the IT division at \ad. Has a 11 year career as system analyst, project manager and IT Director managing external outsourced companies in various industrial and leisure business. Currently uses Web2Py as a quick integration tool to quickly adapt to constant changing requirements.