USDA/GIPSA Proficiency Program
Testing for the Presence of Transgenic Events in Corn and Soybeans
October 2004 Final Report
Purpose of USDA/GIPSA
Proficiency Program
Through the USDA/GIPSA Proficiency Program, USDA seeks to improve the overall performance of testing for biotechnology-derived grains and oil seeds. The USDA/GIPSA Proficiency Program helps organizations identify areas of concern and take corrective actions to improve testing accuracy, capability and reliability.
Program Description
In February 2003, USDA/GIPSA’s Technical Services Division
expanded the program to offer samples for qualitative or quantitative
analysis. Participants could request
samples for qualitative analysis or quantitative analysis. In this round of the
USDA/GIPSA Proficiency Program one set of samples was used for both qualitative
and quantitative analyses. The samples
were fortified with various combinations and concentrations of transgenic
events, and participants had the choice of providing qualitative or quantitative
results. Scoring of the participant’s
results was done by computing the “percentage of correctly reported transgenic
events” in the samples. Two new biotechnology corn events
commercialized in the
Sample Composition
The corn
samples contained various combinations and concentrations of the following transgenic
events: T25, CBH351, MON810, GA21, E176, Bt11, NK603, TC1507, and MON863; or,
no events (i.e., negative corn sample).
To produce the various concentration levels the appropriate amount of
ground transgenic corn was mixed with non-transgenic corn. Fortified corn samples were prepared with concentrations
ranging from 0.1% to 5% of the transgenic event. The soybean samples were either
non-transgenic soybeans, or fortified to a range of 0.1 % to 3 % of the
glyphosate-tolerant event (CP4-EPSPS).
Each
participant received six corn samples and three soybean samples. Each sample contained 20 grams of ground material.
Program Participants
GIPSA mailed samples to sixty-nine
participants and received results from sixty participants. Participants included organizations
from Africa, Asia, Europe, North America, and
Sixty organizations returned
results by the deadline date:
·
Twenty-six
participants submitted qualitative
only results
·
Three participants
submitted quantitative only results
·
Thirty-one
participants submitted a combination of qualitative
and quantitative results.
In this report, participating
organizations were identified by either a confidential “Participant
Identification Number”, or by name.
Appendix I identifies those organizations who gave GIPSA permission to
list them as participants in the USDA/GIPSA Proficiency Program.
Data Summary Results
Data submitted by the participants are summarized in this report primarily in tables and figures. Participants reported their results on a qualitative basis, quantitative basis, or a combination of both types. The quantitative results were qualitatively scored (i.e., on the basis of “incorrect” or “correct’ for the presence or absence of a transgenic event in an unknown sample) instead of being scored on the basis of how accurate their reported quantification value was to the gravimetric fortification level. Quantitative results were reported as the concentration of a particular event in the sample, but were scored qualitatively—either correct or incorrect for a specified event. Participants that reported results “less than” the limit of detection of their analytical method (e.g., < 0.05%) for non-fortified (negative) corn and soybean samples were assessed a “correct” result on those relevant samples.
Qualitative Data Summaries. This section summarizes qualitative sample analysis data:
·
Table One. Performance scores for participants sorted by
event (DNA-based assays).
·
Figure One. Summary of qualitative results for each event
combined with the total number of results (inset values) submitted for the
event (DNA-based testing using conventional PCR).
·
Table Two. Percentage correct scores for all
participants by event based on lateral flow strip testing (Protein-based assays).
·
Table Three. Percentage correct scores for all participants
by event based on ELISA (Enzyme-Linked Immunosorbent Assay testing; Protein-based
assays).
·
Table Four. Error analysis: Percentage of false negative
and false positive results in quantitative analysis results using conventional
PCR, lateral flow strips, and ELISA.
Quantitative Data Summaries. This section summarizes quantitative sample analysis data:
·
Table Five. Performance scores for participants sorted by
event (DNA-based assays).
·
Figure Two. Summary of quantitative results for each event
combined with the total number of results (inset values) submitted for the
event (DNA-based testing using conventional PCR).
·
Table Six. Error analysis: percentage of false negative
and false positive results in quantitative analysis results using real-time
PCR.
·
Table Seven. Summary of participant’s results relative to gravimetric
fortification levels.
Qualitative
Analysis Results
Table One: Performance scores for participants sorted by
event (DNA-based assays).
Figure One: Summary of
qualitative results for each event combined with the total number of results
(inset values) submitted for that particular event (DNA-based testing using
conventional PCR).
Table Two: Percentage correct scores for participants
based on lateral flow strip testing (Protein-based assays).
Table Three: Percentage correct scores based on Enzyme-Linked
Immunosorbent Assay testing (Protein-based assays).
Table Four: Error analysis: percentage of false negative
and false positive results in qualitative analysis results using conventional
PCR, lateral flow strips, and ELISA.
% False Positive = Total number
of incorrect positive reported samples
Total number of
negative samples provided
% False Negative = Total number
of incorrect negative reported samples
Total number
of positive samples provided
Quantitative
Analysis Results
Table Five: Performance scores for
participants sorted by event (DNA-based assays).
Figure Two. Summary of quantitative results
for each event combined with the total number of results (inset values)
submitted for the event (DNA-based testing using real-time PCR).
Table Six: Error analysis: Summary of percentage of
false negative and false positive results in quantitative analysis results
using real-time PCR.
% False Positive = Total number
of incorrect positive reported samples
Total number of
negative samples provided
% False Negative = Total number
of incorrect negative reported samples
Total number
of positive samples provided
Table Seven.
Summary of participant’s results
relative to event fortification levels.
*Indicates that
some values were excluded from analysis because they were outliers (i.e.,
participant’s reported quantifications were > 20x above the fortification
level).
Summary
Qualitative
Analysis Results
For DNA-based testing using conventional PCR, the participants scored 100%
correct on three of twelve events and nearly perfect (≥ 95% correct) on
five of twelve events; however, the event that evinced the lowest performance
was T-25 (Figure One). The participants
were unable to detect 0.1% to 0.4% T-25 in their fortified corn samples about
10% of the time. An analysis of the
source of their collective error showed that the lower performance scores for T-25
were due mainly to the reporting of false negative results (Table Four). This observation of apparent “reduced
detection efficiency” for the T-25 insert has been noted in previous results of
the proficiency testing program (e.g., April 2004 Final Report).
For protein-based testing, using lateral flow strips, participants
achieved a 100% correct score whenever samples were analyzed for CBH351 (Table Two); however, when samples were analyzed for T-25 and
Cry1Ab a mixture of false positives and/or false negatives were observed (Table
Four). In a previous sampling round
(April 2004), similar lower performance scores were observed for those two
events when analyzed by lateral flow strips.
When participants used ELISA, they achieved perfect results for the
events they chose to analyze--with an exception for TC1507 (Table Three) which
identified two false positive results (Table Four).
Quantitative
Analysis Results
The quantitative results were qualitatively scored (i.e., on the basis
of “incorrect” or “correct’ for the presence or absence of a transgenic event
in an unknown sample) instead of being scored on the basis of how accurate the
reported value was to the fortification level.
With GIPSA’s lenient scoring guidelines for DNA-based testing, using
real-time PCR, the participants scored a perfect 100% correct on six of twelve
events and nearly perfect (≥ 95% correct) on all other events with the
exception of MON863 (Figure Two). The reported
errors in sample analysis for MON863 were due entirely to
false positives (Table Four). When comparing October 2004 quantitative results
with that of previous rounds of proficiency test results (i.e., April 2004),
the participants achieved a perfect score on nine of the twelve events—
including MON863. When participants
report results based on the use of real-time PCR (RT-PCR) methods, there are a
greater number of perfect performance scores achieved per event compared with
laboratories reporting qualitative results using conventional PCR methods. We suggest
that better performance scores were achieved using RT- PCR as compared to conventional
PCR because: 1) the RT-PCR method has a lower LOD
(greater sensitivity) than conventional PCR and therefore is able to avoid
reporting false negative results on samples that are fortified at very low
levels (e.g., 0.1% w/w biotechnology-derived plant material), 2) laboratories that use RT-PCR may be more
proficient in their methods 3) quantitative results, reported as the
concentration of a particular event in the sample (%w/w), were scored
qualitatively as either “correct” or “incorrect”, regardless of the
fortification level in the sample (e.g., a sample that was fortified at 5.0%
but was reported as 0.1% received a “correct” result for that analysis). Tables Four and Six show that, for example
with T-25, there was a four-fold decrease (a significant decrease when
tested at the 95% confidence level) in the rate of
false negative reported results when RT-PCR was used instead of [or in
conjunction with] conventional PCR.
We prepared the proficiency samples to contain a range of transgenic
events and fortification levels, thus participants who analyzed them were
tasked to report an array of transgenic events and concentration values. The collective results of their analysis are
summarized in Table Seven according to the number of reported results for that
event (N), the group mean, and standard deviation. Additionally, a measure of “precision” or
reproducibility was computed as the “% CV” (s.d. divided by the mean) x 100,
and a measure of the “accuracy” was computed as “% relative error” (reported
quant value minus fortification value, then divide the difference by the
fortification value) x 100. The notable
trends in this dataset were as follows:
- The
participants reported values in this round were generally more accurate
than values reported in the previous round (April 2004). There was not as strong of an inverse
relationship between “% relative error and fortification level” and
between “%CV and fortification level” as had been observed in the previous
sampling round.
- The
inter-laboratory precision was not measurably improved in an overall
manner as compared to the April 2004 results, as there were still a fair
number of imprecise measurements (CV ≥ 25%). This high level of inter-laboratory
variation (i.e., the variance contributed by all participants for a given
transgenic type and concentration level) could be explained by at least
two main factors: 1) heterogeneity
in the sample matrix being tested, and 2) analytical test method
variability between various laboratories.
We suggest that the second factor may contribute the greater effect
on the high levels of variation in reported quantifications. These results
suggest that there would be technical difficulty in distinguishing between
lots of grain having two different transgenic concentrations. For example, if a randomly selected lab
were tasked to distinguish a lot of grain containing MON810 fortified at
the “0.4% level” from a lot of grain with “0.8% MON810” (Table Seven),
then due to the high % CV’s inherent to each mean value there would be
some overlap of MON810 determinations when analyses were performed on these
two different lots of grain. Some
of the determinations made on the 0.4% lot would intersect at the low end
of the variance associated with determinations made on the 0.8% lot; conversely,
there would be some 0.8% quantifications that overlapped on the high end
of the variance associated with determinations made on the 0.4% lot of
grain.
- The events
with the consistently lowest relative errors (i.e., % relative errors
≤ 20%) at all fortification levels were: CBH 351, BT-11, TC1507 and
NK603. Yet, this was only observed
on four of the nine transgenic events, it is apparent that there exists considerable
difficulty with achieving quantifications that are accurate.
- Another possible
trend in this data set was the high number of “under-reported”
quantifications or reported quantifications that were less than the
fortification level as indicated by a (-) sign in front of the listed
relative error values in Table Seven (e.g., MON810, GA-21, and NK603).
·
To compare the deviations between the reported
quantifications from the known fortification levels we computed the “% relative
error”. It is a simple measure of
“accuracy” (Table Seven). If a given lab
desired to know by how many “standard deviation units” their reported
quantification differed from the collective group mean and standard deviation,
they can compute their Z-score from the information given in Table Seven as
follows: Lab Reported Value – Listed
Group Mean, then divide that result by the listed group’s “standard deviation”
for the particular mean. We advise the
use of due caution in interpreting such Z-scores because we cannot definitively
state that reported values were normally distributed—in some cases the group
means were either positively or negatively skewed from gravimetric values.
We wish to know what types of between groups
or within groups comparisons of performance and data analyses are of interest
to our participants; please feel free to contact us as we prepare future
reports.
Future Studies
In one study we are using the participating laboratory’s “continent of
origin” as a blocking variable to see if there are differences in performance
due to geographical location. For this
analysis we are compiling performance data that have been collected since the
inception of the proficiency program (February 2002) so that a comprehensive
and historical view can be developed.
These results will be submitted for publication in the near future. In another study, we want to know if “tenure
in the proficiency program” has any effect on the performance of testing labs.
We are comparing the performance of veteran (experienced) labs with novice labs
(first-time participants) for a given biotechnology-derived event and
concentration group.
Note: It is important to understand that there are no internationally recognized standard reference materials for all transgenic events. The transgenic seed or grain used to prepare these samples was made available to GIPSA by the Life Science Organizations. Care was taken to ensure the transgenic material was either essentially 100 % positive for the event, or adjusted accordingly. The fortified samples were prepared using a process that has been verified to produce homogenous mixes, and representative samples were analyzed to ensure proper fortification and homogeneity.
To obtain additional information on the
USDA/GIPSA Proficiency Program, contact Dr. Ron Jenkins, USDA/GIPSA Proficiency
Program Manager, at US 816-891-04442, or by e-mail at biotech-lab@usda.gov.
List of Participants: October 2004
These laboratories
declared that they wished to be cited as Participants in the Proficiency
Program. Other labs were in the program,
but expressly wished their identity to remain confidential.
|
Dr. F. Bois |
|
A. Bio. C – Molecular Biology Division |
|
Route de Samadet |
|
64410 ARZACQ |
|
|
|
David Tomas |
|
AINIA |
|
Benjamin Franklin 5-11 |
|
Parque Tecnologico |
|
46980 Paterna |
|
|
|
|
|
Imma Folch |
|
Applus Agroalimentario |
|
Crta. Cabrils, s/n |
|
08348-CABRILS |
|
|
|
|
|
Juan J. Giorda |
|
|
|
Bolsa de Comercio de Rosario |
||
|
Córdoba 1402- 2oPiso |
|
|
|
|
||
|
|
|
|
|
Peter Brodmann |
|
Biolytix AG |
|
Benkenstrasse 254 |
|
CH-4108 Witterswil |
|
|
|
Raquel Cuellar |
|
BIOTOOLS B&M Labs |
|
Valle de Tobalina |
|
52 nave 43 |
|
28021 |
|
|
|
Bureau of Food and Drug Analysis (BFDA), DOH, |
|
|
National Laboratory of Foods and Drugs, Department of |
|
|
161-2, kuen |
|
|
Nankang |
|
|
|
|
|
Bureau of Quality and Safety of Food Department of Medical Sciences |
|
|
|
|
|
Amphur Muang |
|
|
Nonthaburi 11000 |
|
|
|
|
|
|
|
|
Parm Randhawa |
|
|
|
7877 Pleasant |
|
|
|
Cheryl Dollard |
|
Canadian Food Inspection Agency |
|
|
|
|
|
|
|
K2H 8P9 |
|
|
|
Blanca Jauregui, Ph.D. |
|
CNTA-Laboratorio |
|
Ctra N-134 km 50 |
|
31570 San Adrian |
|
Navarra |
|
|
|
Dr. Lutz Grohmann |
|
CONGEN Biotechnology GmbH |
|
Robert Roessle Str. 10 |
|
13125 |
|
Apiwan Yoojinda |
|
DNA Technology Laboratory |
|
Kamphaengsaean, Nakorn Pathon 73140 |
|
|
|
Marco Mazzara |
|
European Commission - JRC |
|
Via Fermi 1 |
|
I-21020 Ispra (VA) |
|
|
|
Dr. Satoshi Futo |
|
FASMAC CO., LTD |
|
|
|
Kanagawa 243-0041 |
|
|
|
Dr. Antje Dietz-Pfeilstetter |
|
Federal Biological Research Centre for Agriculture and Forestry |
|
Institute for Plant Viology, Microbiology and Biosafety |
|
Messeweg 11-12 |
|
D-38104 Braunschweig |
|
|
|
Dr. Gabriele Muecher |
|
GEN-IAL GmbH |
|
Muelheimerstr. 26, Tor 3, Geb. 159 |
|
D-53840 |
|
|
|
Dr. Castor Menendez |
|
GeneScan Analytics GmbH, |
|
Engesserstr. 4 |
|
79108 |
|
|
|
Flavia Machado |
|
GeneScan do Brasil Ltda |
|
Avenida Antonia Gazzola, 1001 |
|
3 andar |
|
13.301-245 ITU - SP
- |
|
Dr. Frank Spiegelhalter |
|
GeneScan |
|
|
|
|
|
Jane Pappin/Bernd Schoel |
|
Genetic ID NA |
|
|
|
|
|
Mariangela Marudelli |
|
Istituto di Botanica e Genetica vegetale |
|
Facoltà di Agraria |
|
Università Cattolica Sacro Cuore |
|
Via Emilia parmense, 84 |
|
29100 |
|
Celine Beinchet |
|
IdentiGen |
|
Unit 9, |
|
|
|
|
|
|
|
Dr. Reinhard Baier |
|
JenaGen Diagnostik-Gentechnik-Biotechnologie |
|
Loebstedter Str. 78 |
|
D-07749 |
|
|
Gerda Hempel
Landesuntersuchungsanstalt fur
das Gesundheits-Veterinarwesen Sachsen
Sitz Dresden
Amtliche Lebensmitteluberwachung
Fachgebiet 6.6
Jagerstrabe 10
D-01099
|
Tamara Roustan, Ph.D. |
|
LEM Laboratoires |
|
Department Biologie moleculaire |
|
38 rue de l’industrie |
|
BP 70192 |
|
67405 Illkirch Cedex |
|
|
|
Dr. Diana Hormisch |
|
LUFA |
|
Obere Langgasse 40 |
|
D-67346 |
|
|
|
Dr. Brigitte Roth |
|
LUFA Augustenberg |
|
D 76227 |
|
Nesslerstr. 23 |
|
|
|
Filippo Odasso |
|
Laboratorio CHMICO CCIAA |
|
Via Vettimiglia
165 |
|
10127 |
|
Dr. Martino Barbanera/ Dr.
Sonia Scaramagli |
|
Laboratorio COOP ITALIA |
|
Via del Lavoro
6/8 |
|
40033 Casalecchio
di Reno |
|
|
|
John C. Jackson, Ph.D. |
|
Monsanto |
|
Q4A |
|
|
|
|
|
Kalyn Brix-Davis |
|
Mid-West Seed Services |
|
|
|
|
|
Dr Gatti Marcello |
|
NEOTRON |
|
NEOTRON Spa Stradello Aggazzotti |
|
104 Santa Maria di Mugnano |
|
Dr. Jana Zel |
|
National |
|
Vecna pot 111 |
|
1000 |
|
|
Ming Zhang
National Testing
No.6 xixinghua street,
gongzhuling
city,
|
Merike Kelve |
|
National |
|
Laboratory of Molecular Genetics |
|
Akadeemia tee 23, |
|
|
|
Ms Kumi Goto |
|
Nippon Yuryo Kentei Kyokai |
|
Bankokubashi Bldg |
|
Kaigan-dori Naka-ku |
|
231-0002, |
|
|
|
Dr. Farin Hajar |
|
OMIC USA Inc |
|
|
|
|
|
Dr. |
|
Pioneer Hi-Bred |
|
|
|
|
Dave Goins
Q Laboratories, Inc.
|
Andrew P Tingey, PhD. |
|
Reading Scientific Services Ltd. |
|
The Lord Zuckerman Research
Centre |
|
Whiteknights |
|
Reading RG66LA |
|
|
|
Dr. |
|
Superinspect Ltda. |
|
Rua do Comercio, 83 |
|
11010-141 Centro |
|
|
|
|
|
Angela Pérez Pérez |
|
Sistemas Genomicos S.L. |
|
|
|
|
|
E-46980 Paterna |
|
|
|
PeO Gummeson/Anders
Dahlqvist |
|
|
|
|
|
SE-230 53 Alnarp |
|
|
|
Maria Saldanha |
|
SGS do Brasil Ltda. |
|
Av. Vereador Alfredo das Neves, 480 |
|
Alemoa |
|
11095-510 |
|
|
|
Dr. Daniel Wetsch |
|
Silliker, Inc. |
|
|
|
|
|
Cristhiane Abegg Bothona |
|
Syngenta Seeds Ltda |
|
BR 452 Km 142 |
|
Uberlandia-MG |
|
|
|
38405-232 |
|
Annelis-Reanate
Winterstein |
|
Thuringer Landesamt fur Lebensmittelsicherheit und Verbraucherschutz |
|
Sitz Jena |
|
Amtliche Lebensmitteluberwachung |
|
Nauburger Str. 96 b |
|
D-07743 |
|
|
Dr. Dahlia Garwe
Tobacco Research Board
Kutsaga Station
Airport Ring Road
Zimbabw
|
Boyce Butler |
|
Thionville Surveying Company |
|
|
|
|
|
Dr. Daniela Contri |
|
TECAM |
|
Rua Fabia, 59 |
|
Sao Paulo – SP – CEP: 05051-030 |
|
|
|
Marc Rindal |
|
Environmental Protection Agency |
|
|
|
|
|
Dr. Juergen Schwendinger |
|
Zentrales Institut des Sanitatsdienstes der Bundeswehr Munchen |
|
Ingolstadter Landstrasse 102 |
|
85748 Garching-Hochbruck |
|
Postfach 45 06 43 |
|
80906 Munchen |
|
|