------- Blind-Carbon-Copy
To: uai06evaluation
Subject: [UAI] Extended Deadlines: Call for Participation: Evaluation of
Probabilistic Inference Systems at UAI06
Reply-To: Jeff Bilmes (206/221-5236) <[EMAIL PROTECTED]>
X-Mailer: MH-E 7.82; nmh 1.0.4; GNU Emacs 21.3.1
Date: Tue, 18 Apr 2006 19:20:55 -0700
From: Jeff Bilmes <[EMAIL PROTECTED]>
[Note: extended deadlines below].
***********************************************************************
CALL FOR PARTICIPATION
Evaluation of Probabilistic Inference Systems
at the
22nd Conference on Uncertainty in Artificial Intelligence (UAI)
July 13th - July 16th, 2006
Massachusetts Institute of Technology, Cambridge, MA, USA
http://ssli.ee.washington.edu/~bilmes/uai06InferenceEvaluation
***********************************************************************
Over the years, many different algorithms for making queries of
probability distributions represented by graphical models have been
developed. There are now a number of algorithms for exact inference
(e.g., junction-tree based schemes using graph triangulations, search
or conditioning based approaches based on SAT/CSP techniques), and
approximation methods (including variational, generalized belief
propagation, and sampling schemes). Also, aspects of software
implementation, such as relevant data structures, memory management,
and subsidiary algorithms, can have a significant effect on actual
runtime.
The purpose of this evaluation is to compare the performance of a
variety of different software systems on a single set of Bayesian
network (BN) problems. A number of different Bayesian networks will be
made available to each participant, and each will be evaluated
according to the following:
1. Exact inference only. While approximate inference is crucial
in many problem domains, recent results have shown that exact
inference can still be performed on a variety of interesting
(and thought previously to be intractable) problems. Therefore,
this first UAI competition will involve systems that perform
exact inference only (meaning everyone's probability score
should be numerically identical).
2. Evaluation will involve both wall-clock run time, and
amount of memory used. Both fastest time with a memory ceiling,
and smallest memory with a time limit will be evaluated.
3. Several forms of query will be possible to evaluate, including
Probability(evidence) (i.e., summing out hidden variables), MPE,
and MAP. The resulting scores/assignments for all queries must be
correct and accurate.
4. A set of complete Bayesian network models will be made available.
Some of the BNs will be strictly positive (meaning there will
be no zeros in the CPTs). Other BNs will be mixtures
of dense, sparse, and deterministic CPTs.
5. We will evaluate only single threaded performance (i.e., no
multi-threading or parallelism in this first evaluation). We
will use the same underlying (Intel-class) microprocessor running
the Linux operating system for all systems. Each process will
have a 2GB memory limitation.
6. Two types of BNs will be evaluated, static and dynamic
(as in dynamic Bayesian networks). They will be placed in a
standard easy-to-read format (to be determined).
We encourage each participant (or any interested party) to submit a
BN of their own to be considered for evaluation (but we do not make
guarantees that any particular BN will be used). All the resulting
BNs used in the competition, and all results, will be listed on the
UAI web after the competition is over (this will encourage continued
standardized evaluation).
IMPORTANT DATES:
5/1/06 - Deadline date to express your intent to participate. You
should
also indicate which of Pr(evidence), MPE and/or MAP you wish
to
evaluate.
5/15/06 - Deadline date to submit a BN and query type for consideration
for
participation. Not all submissions will necessarily be used.
5/30/06 - Date that final list of BNs, their format, and timing
details will be made available.
6/12-23/06 - Evaluation period, during this time, participants will be
given
access to a compute server on which they may compile and
evaluate
their system.
6/23/06 - Final results are due, along with verification script.
7/14/06 - Evaluation results announced at UAI'06
For more and updated information, please visit:
http://ssli.ee.washington.edu/~bilmes/uai06InferenceEvaluation
*--------------------------------------------------------------------*
*--------------------------------------------------------------------*
------- End of Blind-Carbon-Copy
_______________________________________________
uai mailing list
uai@ENGR.ORST.EDU
https://secure.engr.oregonstate.edu/mailman/listinfo/uai
