Important: Scala version of gnparser is entering an 'archive' state. Use https://github.com/gnames/gnparser
We do not have a Scala expert in our group anymore, therefore we ported gnparser
to Go. We strongly recommend to switch to new version and submit bugs/suggestions according to its CONTRIBUTING doc.
Note: To submit issues you can login to GitLab with your GitHub account.
If you have means and a desire to continue development of Scala version please do so!
gnparser
splits scientific names into their component elements with associated meta information. For example, "Homo sapiens Linnaeus"
is parsed into human readable information as follows:
Element | Meaning | Position |
---|---|---|
Homo | genus | (0,4) |
sapiens | specificEpithet | (5,12) |
Linnaeus | author | (13,21) |
Try it as a web-page:
http://scala.parser.globalnames.org
Try it as an web-service:
http://scala.parser.globalnames.org/doc/api
Try it as a command line tool under Linux/Mac:
wget https://github.com/GlobalNamesArchitecture/gnparser/releases/download/release-1.0.2/gnparser-1.0.2.zip
unzip gnparser-1.0.2.zip
sudo rm -rf /opt/gnparser
sudo mv gnparser-1.0.2 /opt/gnparser
sudo rm -f /usr/local/bin/gnparser
sudo ln -s /opt/gnparser/bin/gnparser /usr/local/bin
gnparser name "Homo sapiens Linnaeus"
gnparser --help
gnparser
is also dockerized:
docker pull gnames/gnparser
# to run web-server
docker run -p 80:4334 --name gnparser gnames/gnparser web
# to run socket server
docker run -d -p 4334:4334 --name gnparser gnames/gnparser socket
Finally, run it right from your SBT console:
$ mkdir -p project
$ echo 'sbt.version=0.13.12' > project/build.properties
$ sbt ';set libraryDependencies += "org.globalnames" %% "gnparser" % "1.0.2";console'
scala> import org.globalnames.parser.ScientificNameParser.{instance => scientificNameParser}
scala> scientificNameParser.fromString("Homo sapiens Linnaeus").renderCompactJson
Contents of this Document
Global Names Parser or gnparser
is a Scala library for breaking up scientific names into their different elements. The elements are classified. It is based on parboiled2 -- a Parsing Expression Grammar (PEG) library. The gnparser
project evolved from another PEG-based scientific names parser --biodiversity written in Ruby. Both projects were developed as a part of Global Names Architecture.
Many other parsing algorithms for scientific names use regular expressions. This approach works well for extracting canonical forms in simple cases. However, for complex scientific names and to parse scientific names into all semantic elements regular expressions often fail, unable to overcome the recursive nature of data embedded in names. By contrast, gnparser
is able to deal with the most complex scientific name-strings.
gnparser
takes a name-string like Drosophila (Sophophora) melanogaster Meigen, 1830
and returns parsed components in JSON format. This parser's behavior is defined in its tests and the test file is a good source of information about parser's capabilities, its input and output.
Millions of names parsed per hour on a i7-4930K CPU (6 cores, 12 threads, at 3.4 GHz), parser v0.3.1
Threads | Millions/hr |
---|---|
1 | 29.44 |
2 | 50.85 |
4 | 90.45 |
8 | 120.75 |
12 | 130.9 |
- Fast (~8x faster than biodiversity gem), rock solid and elegant
- Extracts all elements from a name, not only canonical forms
- Works with very complex scientific names, including hybrids
- Can be used directly in any language that can call Java -- Scala, Java, R, Python, Ruby etc.
- Can run as a command line application
- Can run as a socket server
- Can run as a web server
- Can be integrated into Apache Spark-based projects
- Can be scaled to many CPUs and computers
- Calculates a stable UUID version 5 ID from the content of a string
Canonical forms of a scientific name are the latinized components without annotations, authors or dates. They are great for matching names despite alternative spellings. Use the canonicalName -> value
field from parsing results for this use case.
In botany infraspecific ranks play an important role. Use canonicalName -> valueRanked
field to preserve them.
There are many inconsistencies in how scientific names may be written. Use normalized
field to bring them all to a common form (spelling, spacing, ranks).
Many data administrators store name-strings in two columns and split them into "name part" and "authorship part". This practice misses some information when dealing with names like "Prosthechea cochleata (L.) W.E.Higgins var. grandiflora (Mutel) Christenson". However, if this is the use case, a combination of canonicalName -> valueRanked
with the authorship from the lowest taxon will do the job. You can also use --format simple
flag for gnparse
command line tool.
If there are problems with parsing a name, parser generates qualityWarnings
messages and lowers parsing quality
of the name. Quality values mean the following:
"quality": 1
- No problems were detected"quality": 2
- There were small problems, normalized result should still be good"quality": 3
- There were serious problems with the name, and the final result is rather doubtful"parse": false
- A string could not be recognized as a scientific name
gnparser
uses UUID version 5 to generate its id
field. There is algorithmic 1:1 relationship between the name-string and the UUID. Moreover the same algorithm can be used in any popular language to generate the same UUID. Such IDs can be used to globally connect information about name-strings or information associated with name-strings.
More information about UUID version 5 can be found in the Global Names blog.
You can also use UUID calculation library in your code as it is shown in Scala example section.
gnparser
tries to correct problems with spelling, but sometimes it is important to keep original spelling of the canonical forms or authorships. The positions
field attaches semantic meaning to every word in the original name-string and allows users to create canonical forms or other combinations using the original verbatim spelling of the words. Each element in positions
contains 3 parts:
- semantic meaning of a word
- start position of the word
- end position of the word
For example ["specificEpithet", 6, 11]
means that a specific epithet starts at 6th character and ends before 11th character of the string.
The artifacts for gnparser
are located on Maven Central and can be set as a dependency in following ways:
SBT:
libraryDependencies += "org.globalnames" %% "gnparser" % "1.0.2"
Maven:
<dependency>
<groupId>org.globalnames</groupId>
<artifactId>gnparser_2.11</artifactId>
<version>1.0.2</version>
</dependency>
<dependency>
<groupId>org.globalnames</groupId>
<artifactId>gnparser_2.10</artifactId>
<version>1.0.2</version>
</dependency>
Release package should be sufficient for all usages but development. It is not needed for including gnparser
into Java or Scala code -- declare dependency instead.
Java Run Environment (JRE) version >= 1.6 (>= 1.8 for runner project)
File | Description |
---|---|
|
|
release-1.0.2.zip |
Source code's zip file |
release-1.0.2.tar.gz |
Source code's tar file |
wget https://github.com/GlobalNamesArchitecture/gnparser/releases/download/release-1.0.2/gnparser-1.0.2.zip
unzip gnparser-1.0.2.zip
sudo rm -rf /opt/gnparser
sudo mv gnparser-1.0.2 /opt/gnparser
sudo rm -f /usr/local/bin/gnparser
sudo ln -s /opt/gnparser/bin/gnparser /usr/local/bin
- Download gnparser-1.0.2.zip
- Extract it to a place where you usually store program files
- Update your PATH to point to bin subdirectory
- Now you can use
gnparser
command provided bygnparser.bat
script from CMD
Note that gnparser
loads Java runtime environment every time it is called. As a result parsing one name at a time is much slower than parsing many names from a file. When parsing large file expect rates of 6000-9000 name-strings per second on one CPU.
To parse one name
gnparser name "Parus major Linnaeus, 1788"
To parse names from a file (one name per line)
gnparser file --input file_with_names.txt [--output output_file.json --threads 8]
file
is the default command if no command is given. If you want to parse very large number of name-strings using file
subcommand, break data into several files with about 500,000 name-strings in each. To parse names from STDIN to STDOUT:
cat file_with_names.txt | gnparser > file_with_parsed_names.txt
gnparser
accepts the flag --format
(or simply -f
) that determines the output representation. The values are simple
for simple tab-delimited format, json-pretty
and json-compact
for the JSON extended pretty form and the compact form correspondingly
To parse a name-string
gnparser name "Parus major Linnaeus, 1788" --format simple
The simple format returns tab-delimited result with following fields:
- ID of the supplied name-string (generated by UUID5 algorithm)
- Supplied name-string verbatim
- Canonical form without ranks
- Canonical form with ranks
- Authorship (for the lowest rank only)
- Year
- Parsing quality (from 1 to 3, one being the best)
To see help
Note that gnparser -h
shows the JVM
help instead of gnparser
help.
gnparser --help
Use socket (TCP/IP) server when the gnparser
library cannot be imported directly by a programming language. Setting --port
is optional, 4334 is the default port.
gnparser socket --port 1234
To test the socket connection with command line:
telnet localhost 1234
When you see your telnet prompt, enter a name and press Enter
.
To use (TCP/IP) socket server in your code find a socket
library for your language. There is a good chance it is included in the language's core. For example in Ruby it would be:
require "socket"
s = TCPSocket.open("0.0.0.0", 1234)
s.write("Homo sapiens\n")
s.gets
gnparser
TCP server can parse new-line delimited string in a single run. It is more efficient to send several new-line delimited names at once through the socket. gnparser
server returns a string which contains new-line delimited chunks, where each line is a JSON string for a corresponding input name.
Example below also includes a safeguard for "back pressure" cases, where a client application sends strings too fast. TCP server stores data temporarily in buffers before processing, and buffers might get over-filled. At such moment TCP server stops receiving new packets ("back pressure" situation) until it empties its inner queue of messages. Because of that a client application should monitor the count of sent bytes:
require "socket"
require "json"
socket = TCPSocket.open("0.0.0.0", 4334)
open("names.txt").each_slice(100) do |slice|
text = slice.join
until socket.write(text) == text.bytes.size
puts("Reading of a slice starting with #{slice[0]} failed. Retrying")
str = socket.recv(10) until str.nil?
end
slice.each { puts(socket.gets) }
end
Use web-server as an HTTP alternative to TCP/IP server. Setting --port
is optional, 4334 is the default port. To start web server in production mode on http://0.0.0.0:9000
gnparser web --port 9000
Make sure to CGI-escape name-strings for GET requests. An '&' character needs to be converted to '%26'
GET /api?q=Aus+bus|Aus+bus+D.+%26+M.,+1870
POST /api
with request body of JSON array of strings
require 'json'
require 'net/http'
uri = URI('http://scala.parser.globalnames.org/api')
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
request = Net::HTTP::Post.new(uri, 'Content-Type' => 'application/json',
'accept' => 'json')
request.body = ['Solanum mariae Särkinen & S.Knapp',
'Ahmadiago Vánky 2004'].to_json
response = http.request(request)
Several languages are supported either natively or by running their JVM-based versions. The examples folder provides scientific name parsing code snippets for Scala, Java, Jython, JRuby and R languages.
To avoid declaring multiple dependencies Jython, JRuby and R need a reference gnparser fat-jar.
If you decide to follow examples get the code from the release or clone it from GitHub
Scala example is an SBT subproject. To run it execute the command:
sbt 'examples/runMain org.globalnames.parser.examples.ParserScala'
Calculation of UUID version 5 can be done in the following way:
scala> val gen = org.globalnames.UuidGenerator()
scala> gen.generate("Salinator solida")
res0: java.util.UUID = da1a79e5-c16f-5ff7-a925-14c5c7ecdec5
examples/spark/README.rst describes how to use gnparser
with Scala or Python in Apache Spark projects.
Java example is an SBT subproject. To run it execute the command:
sbt 'examples/runMain org.globalnames.parser.examples.ParserJava'
Jython example requires Jython -- a Python language implementation for Java Virtual Machine. Jython distribution should be installed locally according to instructions.
To run it execute the command:
GNPARSER_JAR_PATH=/path/to/gnparser-assembly-1.0.2.jar \
jython examples/jython/parser.py
R example requires rJava package to be installed. To run it execute the command:
Rscript examples/R/parser.R
JRuby example requires JRuby -- a Ruby language implementation for Java Virtual Machine. JRuby distribution should be installed locally according to instructions.
To run it execute the command:
jruby -J-classpath /path/to/gnparser-assembly-1.0.2.jar \
examples/jruby/parser.rb
- Git
- Scala version >= 2.11
- Java SDK version >= 1.8.0
- SBT >= 0.13.17
git clone https://github.com/GlobalNamesArchitecture/gnparser.git
cd gnparser
If you decide to participate in gnparser
development -- fork the repository and submit pull requests of your work.
The project consists of four parts:
parser
contains core routines for parsing input string. It is a minimalistic library and doesn't contain any reference to any rendering library, such as JSONparser-render
extendsparser
functionality to render inner structures to JSONexamples
contains usage samples for some popular programming languagesrunner
contains code required to runparser
from a command line as a standalone tool, as a TCP/IP server, or as a web serverspark-python
provides facilities to callparser
from Sparkbenchmark
contains performance tests
Command | Description |
---|---|
sbt test |
Runs all tests |
sbt ++2.11.12 test |
Runs all tests against Scala v2.10.6 |
|
Creates fat jars for command line and web |
|
Creates executables for command line and web at |
sbt 'runner/run name "Homo sapiens"' |
Parses single name |
|
Runs the web server in development mode at port 9000 |
sbt 'runner/run socket -p 1234' |
Runs the TCP/IP server at port 1234 |
Sometimes it is beneficial to have a jar that contains everything necessary to run a program. Such a jar would include Scala and all required libraries. In the table above you can see that it can be generated with the command sbt assembly
Prebuilt container image can be found on dockerhub
To install/update container
docker pull gnames/gnparser
To run web server
docker run -d -p 80:4334 --name gnparser gnames/gnparser web
To run socket server
docker run -d -p 4334:4334 --name gnparser gnames/gnparser socket
- Alexander Myltsev http://myltsev.com alexander-myltsev@github
- Dmitry Mozzherin dimus@github
Released under MIT license