A1 WEBSITE SCRAPER CODE

Click on the image below to get an idea: The HTML source code relative to the part of the page where the matrix data is stored. The problem we have is that in the original html files the various numbers and amino-acids letters we need are included into an HTML table, so they are heavily mixed-up with HTML tags that we need to get rid of. We can now proceed to extract the data from those HTML files and write the data, in a cleaner format, in some new text files that we will name as a1-matrix.csv, a2-matrix.csv and a3-matrix.csv. Scraping the scoring matrices data from the HTML files

A1 WEBSITE SCRAPER DOWNLOAD

Into script.php the very first task will be to download the three matrices “raw” HTML files, as they are, into the matrix folder, if they do not exist in this folder already (we do not want to create any unnecessary load to the server from which we are scraping the data, so let us ensure we download the original html pages just once). So for now we will have the following structure: To avoid permissions issues during code execution, grant a 777 permission to both the tscore and matrix directories with chmod (chmod 777 directory_path). The code will be written in a file called “script.php” to be created inside the tscore directory. Let us create somewhere within the path of the web root of our apache installation, a directory for the whole project, called tscore and, inside, a subdirectory called matrix, where the matrix work will be executed and the actual matrices files will be stored.

The scoring matrices for these three alleles can be found at the following URLs: Our application will be limited to the analysis with three HLA alleles, namely HLA-A1, A2 and A3. From this analysis we will be able to write dedicated regular expressions and use those in an appropriate code context so as to retrieve the scoring matrix data and write it to text files.Īnother piece of code will be dedicated to reading those text files and storing the data into appropriate data structures (arrays) that can then be used by the form processing script to assign scores to peptides.

A1 WEBSITE SCRAPER MANUAL

For this example we will adopt a simple strategy based on the manual analysis of the HTML source code of our target pages. Web scraping can be achieved by different techniques. The operation of extracting data we need from an existing web page is called “web scraping”. Downloading the scoring matrices raw HTML files from the original website and saving a local copyĪs a first step, we need to download the data relative to each scoring matrix from the website where they are available. In this section we will concentrate on gathering the scoring matrices data from the original web page where they are shared, through a web scraping operation performed by using PHP and regular expressions.

For each peptide the score will be used to rank the peptide according to the thresholds present in the scoring matrix, thereby assigning the peptide to say, the top 10% binders, the top 5% binders, the top 1% binders etc.Īn output will be generated that will list, for each protein, a list of the peptides whose score ranks them above the threshold selected by the user in the initial web form.Īs you can imagine, there is quite some code to be written to achieve all these functionalities. The processing script will generate all possible 9mers from the input protein and assign each one a binding score to the selected HLA allele, based on the appropriate scoring matrix, as discussed in the previous section. The user will be able to select an HLA (MHC) allele for which the user wishes to calculate the binding score to all the possible 9mers (sequences of 9 amino-acids) present in the input sequence and the threshold above which positive peptides will be shown in the analysis results. This web application will accept a protein sequence or UniProt id as input.

Keywords: web scraping, file parsing, PHP programming, regular expressions, MHC-peptide binding, Immunoinformatics Logical flow of the T-Score application

A1 website scraper