[Qemu-devel] -net PCI bus order reversed since 1.14.0

[Rob Landley]

Until recently, -net options used to add interfaces to linux in the same
order they went on the command line, so the first one you listed on the
qemu command line would become eth0, the second -net became eth1, and so
on.  Now they're added in _reverse_ order, so the _last_ one on the
command line is eth0.

I bisected the behavior change to commit
60c07d933c66c4b30a83b7ccbc8a0cb3df1b2d0e but don't understand why it
changes this behavior.  (Probably making udev do something stupid, but
it happens consistently both before and after...)

Rob

[Qemu-devel] [PATCH uq/master V2] kvm: Add CPUID support for VIA CPU

[BrillyWu]

From: BrillyWu 

When KVM is running on VIA CPU with host cpu's model, the
feautures of VIA CPU will be passed into kvm guest by calling
the CPUID instruction for Centaur.

Signed-off-by: BrillyWu
Signed-off-by: KaryJin
---
 target-i386/cpu.h   |3 +++
 target-i386/cpuid.c |   46 +-
 target-i386/kvm.c   |   15 +++
 3 files changed, 63 insertions(+), 1 deletion(-)

--- a/target-i386/cpu.h 2011-05-09 09:55:48.624885001 +0800
+++ b/target-i386/cpu.h 2011-05-09 09:48:53.704885019 +0800
@@ -721,6 +721,9 @@ typedef struct CPUX86State {
 uint32_t cpuid_ext3_features;
 uint32_t cpuid_apic_id;
 int cpuid_vendor_override;
+/* Store the results of Centaur's CPUID instructions */
+uint32_t cpuid_xlevel2;
+uint32_t cpuid_ext4_features;

 /* MTRRs */
 uint64_t mtrr_fixed[11];
--- a/target-i386/cpuid.c   2011-05-09 09:31:11.754884991 +0800
+++ b/target-i386/cpuid.c   2011-05-09 10:18:46.204885008 +0800
@@ -230,6 +230,9 @@ typedef struct x86_def_t {
 char model_id[48];
 int vendor_override;
 uint32_t flags;
+/* Store the results of Centaur's CPUID instructions */
+uint32_t ext4_features;
+uint32_t xlevel2;
 } x86_def_t;

 #define I486_FEATURES (CPUID_FP87 | CPUID_VME | CPUID_PSE)
@@ -522,6 +525,18 @@ static int cpu_x86_fill_host(x86_def_t *
 cpu_x86_fill_model_id(x86_cpu_def->model_id);
 x86_cpu_def->vendor_override = 0;

+/* Call Centaur's CPUID instruction. */
+if (x86_cpu_def->vendor1 == CPUID_VENDOR_VIA_1 &&
+x86_cpu_def->vendor2 == CPUID_VENDOR_VIA_2 &&
+x86_cpu_def->vendor3 == CPUID_VENDOR_VIA_3) {
+host_cpuid(0xC000, 0, &eax, &ebx, &ecx, &edx);
+if (eax >= 0xC001) {
+/* Support VIA max extended level */
+x86_cpu_def->xlevel2 = eax;
+host_cpuid(0xC001, 0, &eax, &ebx, &ecx, &edx);
+x86_cpu_def->ext4_features = edx;
+}
+}

 /*
  * Every SVM feature requires emulation support in KVM - so we can't just
@@ -855,6 +870,8 @@ int cpu_x86_register (CPUX86State *env,
 env->cpuid_xlevel = def->xlevel;
 env->cpuid_kvm_features = def->kvm_features;
 env->cpuid_svm_features = def->svm_features;
+env->cpuid_ext4_features = def->ext4_features;
+env->cpuid_xlevel2 = def->xlevel2;
 if (!kvm_enabled()) {
 env->cpuid_features &= TCG_FEATURES;
 env->cpuid_ext_features &= TCG_EXT_FEATURES;
@@ -1034,7 +1051,12 @@ void cpu_x86_cpuid(CPUX86State *env, uin
uint32_t *ecx, uint32_t *edx)
 {
 /* test if maximum index reached */
-if (index & 0x8000) {
+if ((index & 0xC00f) == index) {
+/* Handle the Centaur's CPUID instruction. */
+if (index > env->cpuid_xlevel2) {
+index = env->cpuid_xlevel2;
+}
+} else if (index & 0x8000) {
 if (index > env->cpuid_xlevel)
 index = env->cpuid_level;
 } else {
@@ -1256,6 +1278,28 @@ void cpu_x86_cpuid(CPUX86State *env, uin
*edx = 0;
}
 break;
+case 0xC000:
+*eax = env->cpuid_xlevel2;
+*ebx = 0;
+*ecx = 0;
+*edx = 0;
+break;
+case 0xC001:
+/* Support for VIA CPU's CPUID instruction */
+*eax = env->cpuid_version;
+*ebx = 0;
+*ecx = 0;
+*edx = env->cpuid_ext4_features;
+break;
+case 0xC002:
+case 0xC003:
+case 0xC004:
+/* Reserved for the future, and now filled with zero */
+*eax = 0;
+*ebx = 0;
+*ecx = 0;
+*edx = 0;
+break;
 default:
 /* reserved values: zero */
 *eax = 0;
--- a/target-i386/kvm.c 2011-05-09 09:31:04.284884996 +0800
+++ b/target-i386/kvm.c 2011-05-09 09:55:42.984885014 +0800
@@ -492,6 +492,21 @@ int kvm_arch_init_vcpu(CPUState *env)
 cpu_x86_cpuid(env, i, 0, &c->eax, &c->ebx, &c->ecx, &c->edx);
 }

+/* Call Centaur's CPUID instructions they are supported. */
+if (env->cpuid_xlevel2 > 0) {
+env->cpuid_ext4_features &=
+kvm_arch_get_supported_cpuid(env, 0xC001, 0, R_EDX);
+cpu_x86_cpuid(env, 0xC000, 0, &limit, &unused, &unused, &unused);
+
+for (i = 0xC000; i <= limit; i++) {
+c = &cpuid_data.entries[cpuid_i++];
+
+c->function = i;
+c->flags = 0;
+cpu_x86_cpuid(env, i, 0, &c->eax, &c->ebx, &c->ecx, &c->edx);
+}
+}
+
 cpuid_data.cpuid.nent = cpuid_i;

 #ifdef KVM_CAP_MCE

Re: [Qemu-devel] [PATCH v2 1/5] ide: Split qdev "ide-drive" into "ide-hd" and "ide-cd"

[Markus Armbruster]

Gerd Hoffmann  writes:

>   Hi,
>
>> +#define DEFINE_IDE_DEV_PROPERTIES() \
>> +DEFINE_PROP_UINT32("unit", IDEDrive, dev.unit, -1), \
>> +DEFINE_BLOCK_PROPERTIES(IDEDrive, dev.conf),\
>> +DEFINE_PROP_STRING("ver",  IDEDrive, dev.version),  \
>> +DEFINE_PROP_STRING("serial",  IDEDrive, dev.serial)
>
> This can also be done this way:
>
> static Property ide_properties[] = {
> DEFINE_PROP_UINT32("unit", IDEDrive, dev.unit, -1),
> [ ... ]
> DEFINE_PROP_END_OF_LIST(),
> };
>
>> +static IDEDeviceInfo ide_dev_info[] = {
>> +{
>> +.qdev.name= "ide-hd",
>> +.qdev.fw_name = "drive",
>> +.qdev.desc= "virtual IDE disk",
>> +.qdev.size= sizeof(IDEDrive),
>> +.init = ide_hd_initfn,
>
>.qdev.props   = ide_properties,
>
> Works only as long as all devices have exactly the same set (i.e. for
> scsi it wouldn't work as not all devices have the "removable"
> property).
>
> I tend to like this more than the #define.  YMMV, matter of taste.

In general, I prefer shared variables rather to duplication via #define,
too.  But in this case, I went with #define anyway, so we don't have to
change from variable to #define when a device-specific property comes
along, and for consistency with scsi-disk.c (whatever that's worth).

Re: [Qemu-devel] [PATCH uq/master] kvm: Add CPUID support for VIA CPU

[BrillyWu]

 hi jan,
Thank you for your help.I choose to use gmail sending emails,and have send a 
new version of the patch,please check it. thank.

brilly


-Original Message-
From: jan.kis...@web.de [mailto:jan.kis...@web.de] 
Sent: Monday, May 09, 2011 2:14 PM
To: Brilly Wu
Cc: a...@redhat.com; qemu-devel@nongnu.org; Kary Jin
Subject: Re: [Qemu-devel] [PATCH uq/master] kvm: Add CPUID support for VIA CPU

On 2011-05-09 07:42, brill...@viatech.com.cn wrote:
> When KVM is running on VIA CPU with host cpu's model, the feautures of 
> VIA CPU will be passed into kvm guest by calling the CPUID instruction 
> for Centaur.
> 
> Signed-off-by: BrillyWu
> Signed-off-by: KaryJin
> ---
>  target-i386/cpu.h   |3 +++
>  target-i386/cpuid.c |   46
> +-

Still line-wrapped.

If your Exchange is unfixable, use a different mail client or even post via a 
different mail provider. In the latter case, add a "From: BrillyWu 
" as first line to the mail body. That will preserve 
the proper authorship on merge.

Jan

[Qemu-devel] [STABLE PULL] usb storage fix

[Gerd Hoffmann]

  Hi,

This is the usb storage fix cherry-picked into the stable branch.

please pull,
  Gerd

The following changes since commit 56a60dd6d619877e9957ba06b92d2f276e3c229d:

  Version 0.14.1 (2011-05-04 13:50:56 -0500)

are available in the git repository at:
  git://git.kraxel.org/qemu usb.stable.14

Gerd Hoffmann (1):
  usb: mass storage fix

 hw/usb-msd.c |1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

[Qemu-devel] [PATCH] usb: mass storage fix

[Gerd Hoffmann]

Initialize scsi_len with zero when starting a new request, so any
stuff leftover from the previous request is cleared out.  This may
happen in case the data returned by the scsi command doesn't fit
into the buffer provided by the guest.

Signed-off-by: Gerd Hoffmann 
(cherry picked from commit ef0bdf77d7070494692cbccd80c4c8f08c85c240)
---
 hw/usb-msd.c |1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/hw/usb-msd.c b/hw/usb-msd.c
index 76f5b02..d73216e 100644
--- a/hw/usb-msd.c
+++ b/hw/usb-msd.c
@@ -364,6 +364,7 @@ static int usb_msd_handle_data(USBDevice *dev, USBPacket *p)
 DPRINTF("Command tag 0x%x flags %08x len %d data %d\n",
 s->tag, cbw.flags, cbw.cmd_len, s->data_len);
 s->residue = 0;
+s->scsi_len = 0;
 s->scsi_dev->info->send_command(s->scsi_dev, s->tag, cbw.cmd, 0);
 /* ??? Should check that USB and SCSI data transfer
directions match.  */
-- 
1.7.1

Re: [Qemu-devel] [PATCH replacement 1/1] Add documentation for qemu_progress_{init, print}()

[Kevin Wolf]

Am 09.05.2011 17:32, schrieb jes.soren...@redhat.com:
> From: Jes Sorensen 
> 
> Signed-off-by: Jes Sorensen 

Thanks, applied to the block branch.

Kevin

Re: [Qemu-devel] [PATCH] ahci: Fix crashes on duplicate BH registration

[Kevin Wolf]

Am 09.05.2011 17:48, schrieb Jan Kiszka:
> If ahci_dma_set_inactive is called a while there is still a pending BH
> from a previous run, we will crash on the second run of
> ahci_check_cmd_bh as it overwrites AHCIDevice::check_bh. Avoid this
> broken and redundant duplicate registration.
> 
> Signed-off-by: Jan Kiszka 

Thanks, applied to the block branch.

Kevin

Re: [Qemu-devel] [PATCH v2 0/2] qed: Periodically flush and clear need check bit

[Kevin Wolf]

Am 09.05.2011 17:45, schrieb Stefan Hajnoczi:
> This patch marks QED images as clean periodically when it is safe to do so.
> This reduces the chance of having to perform a consistency check at startup.
> Previously we left the image dirty even when it was consistent, therefore
> risking an unnecessary consistency check after crash.
> 
> The first patch stubs out qemu-timer.c functions for qemu-tool.c, which is
> needed since block drivers get built into qemu-img and friends.  In the long
> run having timers in qemu-tool too would be good, but for now they are not
> critical and the stub implementation works.

Thanks, applied both to the block branch.

Kevin

[Qemu-devel] [PATCH 0/2] usb-linux: physical port handling.

[Gerd Hoffmann]

  Hi,

These patches fix and improve the physical port handling in the usb host
driver.  Passthrough of devices connected via usb hub should work better
now.  Also you can specify usb devices for passthrough by physical port
(on the host) now.

please review,
  Gerd

Gerd Hoffmann (2):
  usb-linux: fix device path aka physical port handling
  usb-linux: add hostport property

 usb-linux.c |   47 +++
 1 files changed, 27 insertions(+), 20 deletions(-)

[Qemu-devel] [PATCH 1/2] usb-linux: fix device path aka physical port handling

[Gerd Hoffmann]

The device path isn't just a number.  It specifies the physical port
the device is connected to and in case the device is connected via
usb hub you'll have two numbers there, like this: "5.1".  The first
specifies the root port where the hub is plugged into, the second
specifies the port number of the hub where the device is plugged in.
With multiple hubs chained the string can become longer.

This patch renames devpath to port and makes it a string.   It also
adapts the sysfs parsing code accordingly.  The "info usbhost" monitor
command now prints bus number, (os-assigned) device address and physical
port for each device.

Signed-off-by: Gerd Hoffmann 
---
 usb-linux.c |   38 --
 1 files changed, 20 insertions(+), 18 deletions(-)

diff --git a/usb-linux.c b/usb-linux.c
index 72fe6f5..9543a69 100644
--- a/usb-linux.c
+++ b/usb-linux.c
@@ -62,7 +62,7 @@ struct usb_ctrlrequest {
 uint16_t wLength;
 };
 
-typedef int USBScanFunc(void *opaque, int bus_num, int addr, int devpath,
+typedef int USBScanFunc(void *opaque, int bus_num, int addr, char *port,
 int class_id, int vendor_id, int product_id,
 const char *product_name, int speed);
 
@@ -79,6 +79,7 @@ typedef int USBScanFunc(void *opaque, int bus_num, int addr, 
int devpath,
 #define USBPROCBUS_PATH "/proc/bus/usb"
 #define PRODUCT_NAME_SZ 32
 #define MAX_ENDPOINTS 15
+#define MAX_PORTLEN 8
 #define USBDEVBUS_PATH "/dev/bus/usb"
 #define USBSYSBUS_PATH "/sys/bus/usb"
 
@@ -155,7 +156,7 @@ typedef struct USBHostDevice {
 /* Host side address */
 int bus_num;
 int addr;
-int devpath;
+char port[MAX_PORTLEN];
 struct USBAutoFilter match;
 
 QTAILQ_ENTRY(USBHostDevice) next;
@@ -1092,7 +1093,7 @@ static int usb_linux_get_configuration(USBHostDevice *s)
 char device_name[32], line[1024];
 int configuration;
 
-sprintf(device_name, "%d-%d", s->bus_num, s->devpath);
+sprintf(device_name, "%d-%s", s->bus_num, s->port);
 
 if (!usb_host_read_file(line, sizeof(line), "bConfigurationValue",
 device_name)) {
@@ -1138,7 +1139,7 @@ static uint8_t usb_linux_get_alt_setting(USBHostDevice *s,
 char device_name[64], line[1024];
 int alt_setting;
 
-sprintf(device_name, "%d-%d:%d.%d", s->bus_num, s->devpath,
+sprintf(device_name, "%d-%s:%d.%d", s->bus_num, s->port,
 (int)configuration, (int)interface);
 
 if (!usb_host_read_file(line, sizeof(line), "bAlternateSetting",
@@ -1257,7 +1258,7 @@ static int usb_linux_update_endp_table(USBHostDevice *s)
 }
 
 static int usb_host_open(USBHostDevice *dev, int bus_num,
- int addr, int devpath, const char *prod_name)
+ int addr, char *port, const char *prod_name)
 {
 int fd = -1, ret;
 struct usbdevfs_connectinfo ci;
@@ -1283,7 +1284,7 @@ static int usb_host_open(USBHostDevice *dev, int bus_num,
 
 dev->bus_num = bus_num;
 dev->addr = addr;
-dev->devpath = devpath;
+strcpy(dev->port, port);
 dev->fd = fd;
 
 /* read the device description */
@@ -1655,8 +1656,9 @@ static int usb_host_scan_sys(void *opaque, USBScanFunc 
*func)
 {
 DIR *dir = NULL;
 char line[1024];
-int bus_num, addr, devpath, speed, class_id, product_id, vendor_id;
+int bus_num, addr, speed, class_id, product_id, vendor_id;
 int ret = 0;
+char port[MAX_PORTLEN];
 char product_name[512];
 struct dirent *de;
 
@@ -1672,8 +1674,8 @@ static int usb_host_scan_sys(void *opaque, USBScanFunc 
*func)
 if (!strncmp(de->d_name, "usb", 3)) {
 tmpstr += 3;
 }
-if (sscanf(tmpstr, "%d-%d", &bus_num, &devpath) < 1) {
-goto the_end;
+if (sscanf(tmpstr, "%d-%7[0-9.]", &bus_num, port) < 2) {
+continue;
 }
 
 if (!usb_host_read_file(line, sizeof(line), "devnum", de->d_name)) 
{
@@ -1725,7 +1727,7 @@ static int usb_host_scan_sys(void *opaque, USBScanFunc 
*func)
 speed = USB_SPEED_FULL;
 }
 
-ret = func(opaque, bus_num, addr, devpath, class_id, vendor_id,
+ret = func(opaque, bus_num, addr, port, class_id, vendor_id,
product_id, product_name, speed);
 if (ret) {
 goto the_end;
@@ -1816,7 +1818,7 @@ static int usb_host_scan(void *opaque, USBScanFunc *func)
 
 static QEMUTimer *usb_auto_timer;
 
-static int usb_host_auto_scan(void *opaque, int bus_num, int addr, int devpath,
+static int usb_host_auto_scan(void *opaque, int bus_num, int addr, char *port,
   int class_id, int vendor_id, int product_id,
   const char *product_name, int speed)
 {
@@ -1852,7 +1854,7 @@ static int usb_host_auto_scan(void *opaque, int bus_num, 
int addr, int devpath,
 }
 DPRINTF

[Qemu-devel] [PATCH 2/2] usb-linux: add hostport property

[Gerd Hoffmann]

This patch adds a hostport property which allows to specify the host usb
devices to pass through by bus number and physical port.  This means you
can basically hand over one (or more) of the usb plugs on your host to
the guest and whatever device is plugged in there will show up in the
guest.

Usage:

  -device usb-host,hostbus=1,hostport=1

You can figure the port numbers by plugging in some usb device, then
find it in "info usbhost" and pick bus and port specified there.

Signed-off-by: Gerd Hoffmann 
---
 usb-linux.c |9 +++--
 1 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/usb-linux.c b/usb-linux.c
index 9543a69..5f50767 100644
--- a/usb-linux.c
+++ b/usb-linux.c
@@ -135,6 +135,7 @@ struct ctrl_struct {
 struct USBAutoFilter {
 uint32_t bus_num;
 uint32_t addr;
+char *port;
 uint32_t vendor_id;
 uint32_t product_id;
 };
@@ -1416,6 +1417,7 @@ static struct USBDeviceInfo usb_host_dev_info = {
 .qdev.props = (Property[]) {
 DEFINE_PROP_UINT32("hostbus",  USBHostDevice, match.bus_num,0),
 DEFINE_PROP_UINT32("hostaddr", USBHostDevice, match.addr,   0),
+DEFINE_PROP_STRING("hostport", USBHostDevice, match.port),
 DEFINE_PROP_HEX32("vendorid",  USBHostDevice, match.vendor_id,  0),
 DEFINE_PROP_HEX32("productid", USBHostDevice, match.product_id, 0),
 DEFINE_PROP_END_OF_LIST(),
@@ -1838,6 +1840,9 @@ static int usb_host_auto_scan(void *opaque, int bus_num, 
int addr, char *port,
 if (f->addr > 0 && f->addr != addr) {
 continue;
 }
+if (f->port != NULL && (port == NULL || strcmp(f->port, port) != 0)) {
+continue;
+}
 
 if (f->vendor_id > 0 && f->vendor_id != vendor_id) {
 continue;
@@ -2063,7 +2068,7 @@ void usb_host_info(Monitor *mon)
 dec2str(f->addr, addr, sizeof(addr));
 hex2str(f->vendor_id, vid, sizeof(vid));
 hex2str(f->product_id, pid, sizeof(pid));
-monitor_printf(mon, "Device %s.%s ID %s:%s\n",
-   bus, addr, vid, pid);
+monitor_printf(mon, "Bus %s, Addr %s, Port %s, ID %s:%s\n",
+   bus, addr, f->port ? f->port : "*", vid, pid);
 }
 }
-- 
1.7.1

Re: [Qemu-devel] [PATCH V4 00/10] Qemu Trusted Platform Module (TPM) integration

[Stefan Berger]

On 05/10/2011 12:07 AM, Serge E. Hallyn wrote:

To get this to compile on top of qemu-kvm, I needed the following
patch to force CONFIG_THREAD on so as to define things like
qemu_mutex_lock:

Index: qemu-kvm-tpm/configure
===
--- qemu-kvm-tpm.orig/configure 2011-05-09 21:19:10.920002303 -0500
+++ qemu-kvm-tpm/configure  2011-05-09 21:19:22.150002305 -0500
@@ -3420,6 +3420,7 @@
exit 1
fi
echo "CONFIG_TPM=y">>  $config_host_mak
+  echo "CONFIG_THREAD=y">>  $config_host_mak
fi
  fi


That seems to be qemu-kvm specific.

  Stefan

Re: [Qemu-devel] [PATCH v2 0/5] Split ide-drive and scsi-disk qdevs, and more

[Kevin Wolf]

Am 09.05.2011 11:51, schrieb Markus Armbruster:
> This patch series is about purging the "type hint" from the block
> layer.  My previous series cleaned up improper uses it.  Remaining
> uses are info block and qdevs ide-drive, scsi-disk.
> 
> ide-drive and scsi-disk can either act as disk or as CD drive.  They
> use their drive's type hint to decide between disk and CD.  This is
> unclean.  Disk vs. CD needs to be in qdev, not BlockDriverState,
> because it belongs to the drive's guest part.
> 
> Split them into separate devices for disk and CD.  Keep the old ones
> for backward compatibility.
> 
> Remove the type hint from info block.  Its value is unreliable anyway.
> libvirt doesn't use it.
> 
> I posted v1 quite some time ago.  Since we were working towards a
> release then, we decided to take only the bonus bug fixes (PATCH 1-3),
> and revisit the rest later.  Which has turned out to be "somewhat"
> later than anticpiated.  Sorry about that.

Thanks, applied to the block branch.

Kevin

[Qemu-devel] Virt Tools Survey: What to do about virt-clone

[Richard W.M. Jones]

I've volunteered for the task of fixing virt-clone[0].  There are a
number of bugs which need to be addressed.  Unfortunately the current
virt-clone is broken-by-design since it cannot make changes inside the
guest.

  [0] http://linux.die.net/man/1/virt-clone

The bugs boil down to what Microsoft calls "sysprepping" the clone,
which is to say, removing its existing identity, hostname, ssh host
keys, persistent network rules, host SID and workgroup name (for
Windows).  It's helpful for Linux guests to remove some of this
stuff[1] -- it will make the cloning process smoother.  For Windows
it's absolutely required[2].

  [1] 
https://rwmj.wordpress.com/2010/09/24/tip-my-procedure-for-cloning-a-fedora-vm/
  [2] http://technet.microsoft.com/en-us/library/cc721940%28WS.10%29.aspx

All that virt-clone can do now is to copy the guest and make some
simple changes to the libvirt XML (eg. giving it a new MAC address).
It doesn't even address the sysprepping problem.

The problem with sysprepping is that it's hard to do, and it's
different for every operating system.  I've summarized some of the
techniques below.  Worse than that, for some OSes there are different
levels of sysprepping that an administrator might want; also see
below.

So I'd like feedback from "virt-clone next generation" users:

(a) Is cloning guests useful for you or not?  Often or infrequently?

(b) Do you currently use virt-clone to clone guests?

(c) Do you have a homebrew method to clone guests?  What does it do?

(d) Do you use another tool to clone guests?  (And how is it?)

(e) When you clone a guest, do you "sysprep" it or would you like to?
(Using the term "sysprep" generically here, I mean any sort of
reinitialization for Linux or Windows guests).

(f) How do you feel about a multi-step process?

  virt-clone -> virt-sysprep -> virt-resize (for example)

(g) Have you had other problems with cloning guests?

(h) What have I missed out in this analysis?  What other features have
you missed in virt-clone?

Sysprepping Windows
---

This is a complex, manual process.  We do some steps to automate it in
RHEV.  It's best to read Microsoft's online documentation at
[2][3][4].

  [3] http://support.microsoft.com/kb/302577
  [4] http://blogs.technet.com/b/megand/archive/2005/01/20/357570.aspx

Fedora
--

In theory you can just write a file /.unconfigured in the root, and
Fedora will go through the firstboot process at next boot (it will
reset timezone, root password, netconfig, keyboard, authentication).

Some admins will *not* want all of these things to be reset, and will
want either a lesser degree of unconfiguration, or will want to
control each thing manually.

I'm not totally convinced that this hasn't been broken by systemd
introduction in Fedora 15.

general Linux
-

See [1].

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://et.redhat.com/~rjones/virt-df/

Re: [Qemu-devel] [PATCH V4 00/10] Qemu Trusted Platform Module (TPM) integration

[Serge E. Hallyn]

Quoting Stefan Berger (stef...@linux.vnet.ibm.com):
> On 05/10/2011 12:07 AM, Serge E. Hallyn wrote:
> >To get this to compile on top of qemu-kvm, I needed the following
> >patch to force CONFIG_THREAD on so as to define things like
> >qemu_mutex_lock:
> >
> >Index: qemu-kvm-tpm/configure
> >===
> >--- qemu-kvm-tpm.orig/configure  2011-05-09 21:19:10.920002303 -0500
> >+++ qemu-kvm-tpm/configure   2011-05-09 21:19:22.150002305 -0500
> >@@ -3420,6 +3420,7 @@
> >exit 1
> >fi
> >echo "CONFIG_TPM=y">>  $config_host_mak
> >+  echo "CONFIG_THREAD=y">>  $config_host_mak
> >fi
> >  fi
> >
> That seems to be qemu-kvm specific.

Hm, yeah, I guess it is.  Wonder what the odds are of that actually
working then.  Well, I've just about got this and libtpms packaged,
will hopefully finish up this afternoon and see.

I saw nothing problematic in the patches, but just didn't feel
qualified to send acks based on simple review, so figured I'd be
better off actually testing.

What is your plan regarding libtpms?  Will you be making actual
releases at sf.net at some point?

thanks,
-serge

Re: [Qemu-devel] [PATCH V4 00/10] Qemu Trusted Platform Module (TPM) integration

[Stefan Berger]

On 05/10/2011 07:59 AM, Serge E. Hallyn wrote:

Quoting Stefan Berger (stef...@linux.vnet.ibm.com):

On 05/10/2011 12:07 AM, Serge E. Hallyn wrote:

To get this to compile on top of qemu-kvm, I needed the following
patch to force CONFIG_THREAD on so as to define things like
qemu_mutex_lock:

Index: qemu-kvm-tpm/configure
===
--- qemu-kvm-tpm.orig/configure 2011-05-09 21:19:10.920002303 -0500
+++ qemu-kvm-tpm/configure  2011-05-09 21:19:22.150002305 -0500
@@ -3420,6 +3420,7 @@
exit 1
fi
echo "CONFIG_TPM=y">>   $config_host_mak
+  echo "CONFIG_THREAD=y">>   $config_host_mak
fi
  fi


That seems to be qemu-kvm specific.

Hm, yeah, I guess it is.  Wonder what the odds are of that actually
working then.  Well, I've just about got this and libtpms packaged,
will hopefully finish up this afternoon and see.

I saw nothing problematic in the patches, but just didn't feel
qualified to send acks based on simple review, so figured I'd be
better off actually testing.

I am currently making modifications to the patches to support command 
lines like this one to be in line with other devices:


-tpm type=builtin,path=,id=xyz   -device tpm_tis,id=xyz

Typically with command lines like this Qemu also supports multiple 
devices of the same type. With the TPM this is a bit problematic since 
it would need to support multiple TPMs also in the firmware (aka BIOS). 
So I may support this command line but only allow one TPM.



What is your plan regarding libtpms?  Will you be making actual
releases at sf.net at some point?
I was going to wait for a review of all the patches here on the ml and 
see the code checked in -- until that hasn't happened anything could 
change. So for now I am keeping libtpms in Fedora Rawhide and then was 
going to produce a libtpms-0.5.2 and make it commonly available via 
Fedora, maybe putting a copy of the library on sf.net. I would hold off 
on packaging and distributing it.


   Stefan

Re: [Qemu-devel] [fedora-virt] Virt Tools Survey: What to do about virt-clone

[Chris Lalancette]

On 05/10/11 - 12:56:30PM, Richard W.M. Jones wrote:
> Sysprepping Windows
> ---
> 
> This is a complex, manual process.  We do some steps to automate it in
> RHEV.  It's best to read Microsoft's online documentation at
> [2][3][4].
> 
>   [3] http://support.microsoft.com/kb/302577
>   [4] http://blogs.technet.com/b/megand/archive/2005/01/20/357570.aspx

Note that Oz[1] can do some of this.  There is a side-branch of Oz called
"win2k8customize" that has some of this sysprepping functionality built-in.
It's not on master at the moment because it requires something of a hacked-up
tool called "winexe", but we are in the process of finding a replacement
for "winexe" at the moment.

-- 
Chris Lalancette

[1] http://aeolusproject.org/oz.html

Re: [Qemu-devel] Virt Tools Survey: What to do about virt-clone

[Michal Suchanek]

On 10 May 2011 13:56, Richard W.M. Jones  wrote:

> So I'd like feedback from "virt-clone next generation" users:
>
> (a) Is cloning guests useful for you or not?  Often or infrequently?

I clone guests quite often.

>
> (b) Do you currently use virt-clone to clone guests?

no

>
> (c) Do you have a homebrew method to clone guests?  What does it do?
>
> (d) Do you use another tool to clone guests?  (And how is it?)

I use cp(1)
>
> (e) When you clone a guest, do you "sysprep" it or would you like to?
> (Using the term "sysprep" generically here, I mean any sort of
> reinitialization for Linux or Windows guests).

I don't clone Linux much. I would find a script to goes through the
steps like resetting
/etc/udev/rules.d/70-persistent-net.rules and SSH keys handy. However,
not all needs resetting in all cases. Also this is not specific to
virt-clone, this is just a guest application which could be packaged
for distributions regardless of qemu virt-clone or whatever. Adding an
option to trigger it would be handy I guess (eg. a kernel boot
parameter which could be controlled from outside and also read by an
initscript included with such application).

Also the administrator might pre-configure this sysprepping
application to do the right thing, by default it could be interactive
I guess.

For Windows I uninstall the "ACPI compliant computer" device and run
my sysprep script when I want to run multiple instances of the guest
later. This requires a PS/2 keyboard (or emulation).

I don't think you can find a solution to easily trigger such process
on Windows. If you manage to do the above fully automatically you can
also include a script that checks for a file (or a kernel option of
that is possible in Windows) and triggers the process.

>
> (f) How do you feel about a multi-step process?
>
>  virt-clone -> virt-sysprep -> virt-resize (for example)

It's ok to have multiple tools.

Some people would want to use them separately or from scripts some
would want an UI to run all at once so it's good idea to design the
interface both intelligible and machine-usable.


Thanks

Michal

[Qemu-devel] s390x: change mapping base to allow guests > 2GB

[Christian Borntraeger]

Alex,

the current s390x qemu memory layout is

0x100: guest start
0x8000: qemu binary

which limits the amount of available memory to <2GB.
This patch moves the guest pages to 32GB to not collide with the binary
and to leave some space for the program break of qemu. 

Signed-off-by: Christian Borntraeger 


Make sure that big guests (e.g. 4 GB do not collide with the binary)

--- qemu-kvm.orig/exec.c2011-05-04 09:25:22.411957322 +0200
+++ qemu-kvm/exec.c 2011-05-10 14:45:28.158409982 +0200
@@ -2900,10 +2900,14 @@
 #endif
 } else {
 #if defined(TARGET_S390X) && defined(CONFIG_KVM)
-/* XXX S390 KVM requires the topmost vma of the RAM to be < 256GB 
*/
-new_block->host = mmap((void*)0x100, size,
+/* S390 KVM requires the topmost vma of the RAM to be smaller than
+   an system defined value, which is at least 256GB. Larger systems
+   have larger values. We put the guest between the end of data
+   segment (system break) and this value. We use 32GB as a base to
+   have enough room for the system break to grow. */
+new_block->host = mmap((void*)0x8, size,
PROT_EXEC|PROT_READ|PROT_WRITE,
-   MAP_SHARED | MAP_ANONYMOUS, -1, 0);
+   MAP_SHARED | MAP_ANONYMOUS | MAP_FIXED, -1, 
0);
 #else
 new_block->host = qemu_vmalloc(size);
 #endif

[Qemu-devel] s390x: fix memory detection for guests > 64GB

[Christian Borntraeger]

Alex,

the s390 memory detection has a 16bit field that specifies the amount of
increments. This patch adopts the memory size to always fit into that
scheme. This also fixes virtio detection for these guests, since the 
descriptor page is located after the main memory.

Signed-off-by: Christian Borntraeger 

--- a/target-s390x/op_helper.c
+++ b/target-s390x/op_helper.c
@@ -2361,6 +2361,7 @@ static void ext_interrupt(CPUState *env, int type, 
uint32_t param,
 int sclp_service_call(CPUState *env, uint32_t sccb, uint64_t code)
 {
 int r = 0;
+int shift = 0;
 
 #ifdef DEBUG_HELPER
 printf("sclp(0x%x, 0x%" PRIx64 ")\n", sccb, code);
@@ -2375,8 +2376,10 @@ int sclp_service_call(CPUState *env, uint32_t sccb, 
uint64_t code)
 switch(code) {
 case SCLP_CMDW_READ_SCP_INFO:
 case SCLP_CMDW_READ_SCP_INFO_FORCED:
-stw_phys(sccb + SCP_MEM_CODE, ram_size >> 20);
-stb_phys(sccb + SCP_INCREMENT, 1);
+while ((ram_size >> (20 + shift)) > 65535)
+   shift++;
+stw_phys(sccb + SCP_MEM_CODE, ram_size >> (20 + shift));
+stb_phys(sccb + SCP_INCREMENT, 1 << shift);
 stw_phys(sccb + SCP_RESPONSE_CODE, 0x10);
 
 if (kvm_enabled()) {
diff --git a/vl.c b/vl.c
index 68c3b53..7b4adff 100644
--- a/vl.c
+++ b/vl.c
@@ -2962,6 +2962,16 @@ int main(int argc, char **argv, char **envp)
 if (ram_size == 0)
 ram_size = DEFAULT_RAM_SIZE * 1024 * 1024;
 
+/* s390x ram size detection needs a 16bit multiplier + an increment. So
+   guests > 64GB can be specified in 2MB steps etc */
+if (strstr(machine->name, "s390")) { 
+int shift = 0;
+
+while ((ram_size >> (20 + shift)) > 65535)
+   shift++;
+ram_size = ram_size >> (20 + shift) << (20 + shift);
+}
+
 /* init the dynamic translator */
 cpu_exec_init_all(tb_size * 1024 * 1024);
 

Re: [Qemu-devel] [virt-tools-list] Virt Tools Survey: What to do about virt-clone

[Bruno Lamps]

>
> (a) Is cloning guests useful for you or not?  Often or infrequently?
>

I clone VMs once os twice a month. It's usefull for creating test
environments for the applications this company runs, and to safely backup a
VM before doing some change of update.

(b) Do you currently use virt-clone to clone guests?
>

I use virt-manager. Don't know what command virt-manager uses. I just right
click the VM and clone it.

(c) Do you have a homebrew method to clone guests?  What does it do?


I don't. If I need to clone a VM without virt-manager, I would make copies
of VM hard disk and vm XML, you know, manually copy the stuff.

(d) Do you use another tool to clone guests?  (And how is it?)


No, I don't.

(e) When you clone a guest, do you "sysprep" it or would you like to?
> (Using the term "sysprep" generically here, I mean any sort of
> reinitialization for Linux or Windows guests).
>

I don't. Just change the MAC address (specially in windows guests).

(f) How do you feel about a multi-step process?


I really would enjoy having more options when cloning machine, could save
lots of time in here. I don't resize cloned machines, for example. When I
need to change the size of the cloned machine, I just create a new one from
scratch.

(g) Have you had other problems with cloning guests?
>

If I change the Mac address of the NICs of a linux guest, O.S. can't find
the NIC anymore. It's a problem to me, since I started working with linux
servers about a year ago and I'm still (! lol) not familiar with "manually
adding hardware"...

(h) What have I missed out in this analysis?  What other features have
> you missed in virt-clone?
>

Can't remember any right now... =D

On Tue, May 10, 2011 at 8:56 AM, Richard W.M. Jones wrote:

> I've volunteered for the task of fixing virt-clone[0].  There are a
> number of bugs which need to be addressed.  Unfortunately the current
> virt-clone is broken-by-design since it cannot make changes inside the
> guest.
>
>  [0] http://linux.die.net/man/1/virt-clone
>
> The bugs boil down to what Microsoft calls "sysprepping" the clone,
> which is to say, removing its existing identity, hostname, ssh host
> keys, persistent network rules, host SID and workgroup name (for
> Windows).  It's helpful for Linux guests to remove some of this
> stuff[1] -- it will make the cloning process smoother.  For Windows
> it's absolutely required[2].
>
>  [1]
> https://rwmj.wordpress.com/2010/09/24/tip-my-procedure-for-cloning-a-fedora-vm/
>  [2] http://technet.microsoft.com/en-us/library/cc721940%28WS.10%29.aspx
>
> All that virt-clone can do now is to copy the guest and make some
> simple changes to the libvirt XML (eg. giving it a new MAC address).
> It doesn't even address the sysprepping problem.
>
> The problem with sysprepping is that it's hard to do, and it's
> different for every operating system.  I've summarized some of the
> techniques below.  Worse than that, for some OSes there are different
> levels of sysprepping that an administrator might want; also see
> below.
>
> So I'd like feedback from "virt-clone next generation" users:
>
> (a) Is cloning guests useful for you or not?  Often or infrequently?
>
> (b) Do you currently use virt-clone to clone guests?
>
> (c) Do you have a homebrew method to clone guests?  What does it do?
>
> (d) Do you use another tool to clone guests?  (And how is it?)
>
> (e) When you clone a guest, do you "sysprep" it or would you like to?
> (Using the term "sysprep" generically here, I mean any sort of
> reinitialization for Linux or Windows guests).
>
> (f) How do you feel about a multi-step process?
>
>  virt-clone -> virt-sysprep -> virt-resize (for example)
>
> (g) Have you had other problems with cloning guests?
>
> (h) What have I missed out in this analysis?  What other features have
> you missed in virt-clone?
>
> Sysprepping Windows
> ---
>
> This is a complex, manual process.  We do some steps to automate it in
> RHEV.  It's best to read Microsoft's online documentation at
> [2][3][4].
>
>  [3] http://support.microsoft.com/kb/302577
>  [4] http://blogs.technet.com/b/megand/archive/2005/01/20/357570.aspx
>
> Fedora
> --
>
> In theory you can just write a file /.unconfigured in the root, and
> Fedora will go through the firstboot process at next boot (it will
> reset timezone, root password, netconfig, keyboard, authentication).
>
> Some admins will *not* want all of these things to be reset, and will
> want either a lesser degree of unconfiguration, or will want to
> control each thing manually.
>
> I'm not totally convinced that this hasn't been broken by systemd
> introduction in Fedora 15.
>
> general Linux
> -
>
> See [1].
>
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat
> http://people.redhat.com/~rjones
> virt-df  lists disk usage of
> guests without needing to install any
> software inside the virtual machine.  Supports Linux and Windows.
> http://et.redhat

Re: [Qemu-devel] KVM call agenda for May 10th

[Jes Sorensen]

On 05/09/11 13:50, Juan Quintela wrote:
> 
> Please send in any agenda items you are interested in covering.
> 
> From last week, we have already:
> 
>  - import kvm headers into qemu, drop #ifdef maze (Jan)
> 
> Thanks, Juan.
> 

Since we haven't received any further agenda items. In addition Anthony
is unavailable, plus all the guys in Tel Aviv is on holiday today - lets
postpone until next week.

CANCELED!

Jes

Re: [Qemu-devel] [virt-tools-list] Virt Tools Survey: What to do about virt-clone

[Decker, Schorschi]

(a) Is cloning guests useful for you or not?  Often or infrequently?

Our end-users are quite virtualization aware, they have demanded that we
support a number of features in our virtual infrastructure, this
included.  It will be used frequently and often, maybe 100s of times a
day across our global infrastructure.

(b) Do you currently use virt-clone to clone guests?

Only in the lab, cloning or templating is a new initiative in our
maturing design, so the timing of revamping this is perfect for our
plans over the next few months.

(c) Do you have a homebrew method to clone guests?  What does it do?

Yes, we often migrate and/or clone VMs in the lab between VMware and KVM
and Hyper-V and back again.  VMware and KVM is quite easy, given the
flexibility of common shared storage support and VM disks in RAW format
adaptable to VMware and KVM.  Hyper-V has its own challenges given its
limitations by design.  For VMware to KVM and back, we have a few bash
scripts that we use.  Migrations are straight-forward, since no
personality customization is needed, but true clones, which we use for
SPECVirt analysis for example, we have to remove/reapply personality.
This is not trivial, since we have a lot of applications in our base
image that are infrastructure aware, and require reconfiguration, or
re-initialization, beyond the typical sysprep or OS level changes.
Windows is especially painful on this point.  Our personality management
scripting is in a constant state of change because we are routinely
updating our core images.

(d) Do you use another tool to clone guests?  (And how is it?)

We have used platespin off and on, and of course use various tools and
methods core to WAIK tool set for Windows.  But custom scripting is the
significant 800 pound gorilla in the room on this point.  A specific in
VM agent or service that would explicitly support re-personalization
regardless of OS or application level would be of great benefit, since
meta-data or a back-end database could drive the re-personalization,
provided a more stateless model.  Especially for Windows this might be
an idea worth consideration.  BladeLogic, Ops-ware, Altaris, etc. all
use some form of this idea, but they are also focused on provisioning in
general so the meta-data driven re-personalization gets lost in the
overall product.

(e) When you clone a guest, do you "sysprep" it or would you like to?
(Using the term "sysprep" generically here, I mean any sort of
reinitialization for Linux or Windows guests).

Sysprep is fine as a place to start, but even when Microsoft wrote
sysprep, it was an idea that was ignored for a long time, and it was
never focused on, the world has since gone forward, so sysprep is at
best, only a starting point for what is needed to abstract personality.
A better solution would be that we image/clone the personality specific
deltas and archive them to a library, rather than complete clones that
have personality logic staged for implementation on first boot.

(f) How do you feel about a multi-step process?

  virt-clone -> virt-sysprep -> virt-resize (for example)

As long as any GUI and CLI (as well as API) are consistent in feature
set and flow, multi-step is fine.   We need to be able to completely
automate the process, since we would drive it from a self-serve web
portal concept.

(g) Have you had other problems with cloning guests?

Nothing beyond what was outlined above.  Abstracting of personality at
OS and application level is key for us.

(h) What have I missed out in this analysis?  What other features have
you missed in virt-clone?

No specifically to virt-clone but in general, abstracting the common
code, to common library or binary, so that virt-manager, virsh, etc. all
reference the same design, feature, and executables, is the strategic
direction that should be followed.  The idea that different code is
trying to implement same concepts in different tools makes no long term
strategic sense for the KVM platform in general.


Schorschi Decker

VP; Sr. Consultant Engineer
ET&D Emerging Technologies / Virtualization Platform Engineering Team
Bank of America

-Original Message-
From: virt-tools-list-boun...@redhat.com
[mailto:virt-tools-list-boun...@redhat.com] On Behalf Of Richard W.M.
Jones
Sent: Tuesday, 10 May, 2011 04:57
To: virt-tools-l...@redhat.com; Cole Robinson
Cc: v...@lists.fedoraproject.org; qemu-devel@nongnu.org
Subject: [virt-tools-list] Virt Tools Survey: What to do about
virt-clone

I've volunteered for the task of fixing virt-clone[0].  There are a
number of bugs which need to be addressed.  Unfortunately the current
virt-clone is broken-by-design since it cannot make changes inside the
guest.

  [0] http://linux.die.net/man/1/virt-clone

The bugs boil down to what Microsoft calls "sysprepping" the clone,
which is to say, removing its existing identity, hostname, ssh host
keys, persistent network rules, host SID and workgroup name (for
Windows).  It's helpful for Linux guests to remove s

Re: [Qemu-devel] [RFC] live snapshot, live merge, live block migration

[Marcelo Tosatti]

On Mon, May 09, 2011 at 04:40:00PM +0300, Dor Laor wrote:
> No patch here (sorry) but collection of thoughts about these
> features and their potential building blocks. Please review (also on
> http://wiki.qemu.org/Features/LiveBlockMigration)
> 
> Future qemu is expected to support these features (some already
> implemented):
> 
>  * Live block copy
> 
>Ability to copy 1+ virtual disk from the source backing file/block
>device to a new target that is accessible by the host. The copy
>supposed to be executed while the VM runs in a transparent way.
> 
>Status: code exists (by Marcelo) today in qemu but needs refactoring
>due to a race condition at the end of the copy operation. We agreed
>that a re-implementation of the copy operation should take place
>that makes sure the image is completely mirrored until management
>decides what copy to keep.
> 
>  * Live snapshots and live snapshot merge
> 
>Live snapshot is already incorporated (by Jes) in qemu (still need
>qemu-agent work to freeze the guest FS).
> 
>Live snapshot merge is required in order of reducing the overhead
>caused by the additional snapshots (sometimes over raw device).
>Currently not implemented for a live running guest
> 
>Possibility: enhance live copy to be used for live snapshot merge.
> It is almost the same mechanism.

The idea is to use live block copy to perform snapshot "live merges".
The advantage is the simplicity, since there is no need to synchronize
between live merge writes and guest writes.

With live copy the guest is either using the old image or the new copy,
so crash handling is relatively simple.

>  * Copy on read (image streaming)
>Ability to start guest execution while the parent image reside
>remotely and each block access is replicated to a local copy (image
>format snapshot)
> 
>It should be nice to have a general mechanism that will be used for
>all image formats. What about the protocol to access these blocks
>over the net? We can reuse existing ones (nbd/iscsi).
> 
>Such functionality can be hooked together with live block migration
>instead of the 'post copy' method.
> 
>  * Live block migration (pre/post)
> 
>Beyond live block copy we'll sometimes need to move both the storage
>and the guest. There are two main approached here:
>- pre copy
>  First live copy the image and only then live migration the VM.
>  It is simple but if the purpose of the whole live block migration
>  was to balance the cpu load, it won't be practical to use since
>  copying an image of 100GB will take too long.
>- post copy
>  First live migrate the VM, then live copy it's blocks.
>  It's better approach for HA/load balancing but it might make
>  management complex (need to keep the source VM alive, what happens
>  on failures?)
>  Using copy on read might simplify it -
>  post copy = live snapshot + copy on read.
> 
>In addition there are two cases for the storage access:
>1. The source block device is shared and can be easily accessed by
>   the destination qemu-kvm process.
>   That's the easy case, no special protocol needed for the block
>   devices copying.
>2. There is no shared storage at all.
>   This means we should implement a block access protocol over the
>   live migration fd :(
> 
>   We need to chose whether to implement a new one, or re-use NBD or
>   iScsi (target&initiator)
> 
>  * Using external dirty block bitmap
> 
>FVD has an option to use external dirty block bitmap file in
>addition to the regular mapping/data files.
> 
>We can consider using it for live block migration and live merge too.
>It can also allow additional usages of 3rd party tools to calculate
>diffs between the snapshots.
>There is a big down side thought since it will make management
>complicated and there is the risky of the image and its bitmap file
>get out of sync. It's much better choice to have qemu-img tool to be
>the single interface to the dirty block bitmap data.
> 
> Summary:
>   * We need Marcelo's new (to come) block copy implementation
> * should work in parallel to migration and hotplug
>   * General copy on read is desirable
>   * Live snapshot merge to be implemented using block copy
>   * Need to utilize a remote block access protocol (iscsi/nbd/other)
> Which one is the best?
>   * Keep qemu-img the single interface for dirty block mappings.
>   * Live block migration pre copy == live copy + block access protocol
> + live migration
>   * Live block migration post copy == live migration + block access
> protocol/copy on read.
> 
> Comments?
> 
> Regards,
> Dor

Re: [Qemu-devel] [PATCH V4 00/10] Qemu Trusted Platform Module (TPM) integration

[Serge E. Hallyn]

Quoting Stefan Berger (stef...@linux.vnet.ibm.com):
> >What is your plan regarding libtpms?  Will you be making actual
> >releases at sf.net at some point?
> I was going to wait for a review of all the patches here on the ml
> and see the code checked in -- until that hasn't happened anything
> could change. So for now I am keeping libtpms in Fedora Rawhide and
> then was going to produce a libtpms-0.5.2 and make it commonly
> available via Fedora, maybe putting a copy of the library on sf.net.
> I would hold off on packaging and distributing it.

Ok, thanks, so I certainly won't push it into the archives yet, but
will keep it in a ppa for easy testing.

thanks,
-serge

Re: [Qemu-devel] [PATCH 0/2] usb-linux: physical port handling.

[Brad Hards]

On Tuesday 10 May 2011 12:30:41 Gerd Hoffmann wrote:
>   Hi,
> 
> These patches fix and improve the physical port handling in the usb host
> driver.  Passthrough of devices connected via usb hub should work better
> now.  Also you can specify usb devices for passthrough by physical port
> (on the host) now.
> 
> please review,
>   Gerd
> 
> Gerd Hoffmann (2):
>   usb-linux: fix device path aka physical port handling
>   usb-linux: add hostport property
> 
>  usb-linux.c |   47 +++
>  1 files changed, 27 insertions(+), 20 deletions(-)
Some additional docs (based on 2/2 hints) might be a useful addition.
Possible candidates:
- docs/qdev-device-use.txt
- qemu-doc.texi

Brad

Re: [Qemu-devel] [virt-tools-list] Virt Tools Survey: What to do about virt-clone

[Decker, Schorschi]

Given a couple of questions I have gotten in response to my survey
responses... let me clarify, although I imply we do a significant volume
of V2V, which we do as well, the responses below are targeted and
applicable to cloning.  Especially the need to re-personalize clones
beyond the OS scope to the application scope when and where possible.

We maintain the original VMs in the original environment, often running
clones in parallel on different environments, so cloning is done, both
within the same hypervisor scope and across different hypervisor
environments.  Our work with SPECVirt is one concrete example where we
use cloning routinely, to build multiple tiers or tiles.  Our focus thus
far has been in the lab/test/development space, but our client base is
pushing for same flexibility in QA/production environments, so the need
for better/mature tools and methods in the KVM space, including cloning,
is key to our future goals.


Schorschi Decker

VP; Sr. Consultant Engineer
ET&D Emerging Technologies / Virtualization Platform Engineering Team
Bank of America


-Original Message-
From: Decker, Schorschi 
Sent: Tuesday, 10 May, 2011 07:01
To: virt-tools-l...@redhat.com; Cole Robinson
Cc: v...@lists.fedoraproject.org; qemu-devel@nongnu.org
Subject: RE: [virt-tools-list] Virt Tools Survey: What to do about
virt-clone

(a) Is cloning guests useful for you or not?  Often or infrequently?

Our end-users are quite virtualization aware, they have demanded that we
support a number of features in our virtual infrastructure, this
included.  It will be used frequently and often, maybe 100s of times a
day across our global infrastructure.

(b) Do you currently use virt-clone to clone guests?

Only in the lab, cloning or templating is a new initiative in our
maturing design, so the timing of revamping this is perfect for our
plans over the next few months.

(c) Do you have a homebrew method to clone guests?  What does it do?

Yes, we often migrate and/or clone VMs in the lab between VMware and KVM
and Hyper-V and back again.  VMware and KVM is quite easy, given the
flexibility of common shared storage support and VM disks in RAW format
adaptable to VMware and KVM.  Hyper-V has its own challenges given its
limitations by design.  For VMware to KVM and back, we have a few bash
scripts that we use.  Migrations are straight-forward, since no
personality customization is needed, but true clones, which we use for
SPECVirt analysis for example, we have to remove/reapply personality.
This is not trivial, since we have a lot of applications in our base
image that are infrastructure aware, and require reconfiguration, or
re-initialization, beyond the typical sysprep or OS level changes.
Windows is especially painful on this point.  Our personality management
scripting is in a constant state of change because we are routinely
updating our core images.

(d) Do you use another tool to clone guests?  (And how is it?)

We have used platespin off and on, and of course use various tools and
methods core to WAIK tool set for Windows.  But custom scripting is the
significant 800 pound gorilla in the room on this point.  A specific in
VM agent or service that would explicitly support re-personalization
regardless of OS or application level would be of great benefit, since
meta-data or a back-end database could drive the re-personalization,
provided a more stateless model.  Especially for Windows this might be
an idea worth consideration.  BladeLogic, Ops-ware, Altaris, etc. all
use some form of this idea, but they are also focused on provisioning in
general so the meta-data driven re-personalization gets lost in the
overall product.

(e) When you clone a guest, do you "sysprep" it or would you like to?
(Using the term "sysprep" generically here, I mean any sort of
reinitialization for Linux or Windows guests).

Sysprep is fine as a place to start, but even when Microsoft wrote
sysprep, it was an idea that was ignored for a long time, and it was
never focused on, the world has since gone forward, so sysprep is at
best, only a starting point for what is needed to abstract personality.
A better solution would be that we image/clone the personality specific
deltas and archive them to a library, rather than complete clones that
have personality logic staged for implementation on first boot.

(f) How do you feel about a multi-step process?

  virt-clone -> virt-sysprep -> virt-resize (for example)

As long as any GUI and CLI (as well as API) are consistent in feature
set and flow, multi-step is fine.   We need to be able to completely
automate the process, since we would drive it from a self-serve web
portal concept.

(g) Have you had other problems with cloning guests?

Nothing beyond what was outlined above.  Abstracting of personality at
OS and application level is key for us.

(h) What have I missed out in this analysis?  What other features have
you missed in virt-clone?

No specifically to virt-clone but in general, abstracting the

Re: [Qemu-devel] Virt Tools Survey: What to do about virt-clone

[Gerd Hoffmann]

  Hi,


(a) Is cloning guests useful for you or not?  Often or infrequently?


I'm almost never do that.

Usually I use qcow2 copy-on-write images for testing, so I can easily 
rollback stuff by just zapping and re-creating the copy-on-write image.


When I need a fresh VM I install one.  Have kickstart in place for 
RHEL/Fedora guests.  Also a custom windows xp install cd which 
autoinstalls via unattended.txt, so I don't have to babysit the 
installs.  Includes sp3 and some extra drivers (e1000, virtio-net, ...) too.


cheers,
  Gerd

Re: [Qemu-devel] [PATCH 00/26] q35 chipset support for native pci?express support

[Adnan Khaleel]

I found one problem in the bios config that was preventing the BIOS from seeing 
the MBR. 
The AHCI support was not enabled. To fix this 
change seabios/.config


CONFIG_AHCI=y


The system now loads the boot loader and does some device initialization. 
However, the boot gets stuck when the OS partition cannot be seen and I get a 
message saying 


"Waiting for device /dev/disk/by-uuid/ to appear:  Could not 
find /dev/disk/by-uuid/"


Any ideas whats happening here?


AK
  _  

From: Adnan Khaleel [mailto:ad...@khaleel.us]
To: ad...@khaleel.us
Cc: Hu Tao [mailto:hu...@cn.fujitsu.com]
Sent: Tue, 10 May 2011 11:43:33 -0500
Subject: Re: [Qemu-devel] [PATCH 00/26] q35 chipset support for native 
pci?express support

Hi Hu,


Have you managed to get this working? 


Adnan
  _  

From: Adnan Khaleel [mailto:ad...@khaleel.us]
To: Isaku Yamahata [mailto:yamah...@valinux.co.jp]
Cc: Hu Tao [mailto:hu...@cn.fujitsu.com], qemu-devel@nongnu.org
Sent: Thu, 21 Apr 2011 11:12:37 -0500
Subject: Re: [Qemu-devel] [PATCH 00/26] q35 chipset support for native 
pci?express support

I still get the same error:



akhaleel@depot5 qemu_0.14_q35 $ git clone 
http://people.valinux.co.jp/~yamahata/qemu/q35/20110316/qemu
Getting alternates list for 
http://people.valinux.co.jp/~yamahata/qemu/q35/20110316/qemu/
Getting pack list for 
http://people.valinux.co.jp/~yamahata/qemu/q35/20110316/qemu/
Getting index for pack c0c6d1b7fc8ae79abf99bfb6a402b50e2ec98557
Getting pack c0c6d1b7fc8ae79abf99bfb6a402b50e2ec98557
 which contains ad620c29c2da573e3a5f13f5b1eb2694fee64cfb
error: cannot unpack 000198da6f46c240e46c562caf57b14268d27597 from 
/users/akhaleel/akhaleel/MergeSpace/qemu_0.14_q35/qemu/.git/objects/pack/pack-c0c6d1b7fc8ae79abf99bfb6a402b50e2ec98557.pack
:
:

error: cannot unpack fffd440d2ca664a03ee83eabc00107eaf74d7af4 from 
/users/akhaleel/akhaleel/MergeSpace/qemu_0.14_q35/qemu/.git/objects/pack/pack-c0c6d1b7fc8ae79abf99bfb6a402b50e2ec98557.pack
error: Unable to find ad620c29c2da573e3a5f13f5b1eb2694fee64cfb under 
http://people.valinux.co.jp/~yamahata/qemu/q35/20110316/qemu/
Cannot obtain needed none ad620c29c2da573e3a5f13f5b1eb2694fee64cfb
while processing commit .
rm: cannot remove directory 
`/users/akhaleel/akhaleel/MergeSpace/qemu_0.14_q35/qemu/.git/clone-tmp': 
Directory not empty

  _  

From: Isaku Yamahata [mailto:yamah...@valinux.co.jp]
To: Adnan Khaleel [mailto:ad...@khaleel.us]
Cc: Hu Tao [mailto:hu...@cn.fujitsu.com], qemu-devel@nongnu.org
Sent: Wed, 20 Apr 2011 21:07:46 -0500
Subject: Re: [Qemu-devel] [PATCH 00/26] q35 chipset support for native 
pci?express support

Okay. Can you please try git clone again?
  
  On Wed, Apr 20, 2011 at 06:41:56PM -0500, Adnan Khaleel wrote:
  > Something is still wrong,
  > 
  > I get the following errors now:
  > 
  > :
  > error: cannot unpack fffd440d2ca664a03ee83eabc00107eaf74d7af4 from /users/
  > akhaleel/akhaleel/MergeSpace/qemu_0.14_q35/qemu/.git/objects/pack/
  > pack-c0c6d1b7fc8ae79abf99bfb6a402b50e2ec98557.pack
  > error: Unable to find ad620c29c2da573e3a5f13f5b1eb2694fee64cfb under http://
  > people.valinux.co.jp/~yamahata/qemu/q35/20110316/qemu/
  > Cannot obtain needed none ad620c29c2da573e3a5f13f5b1eb2694fee64cfb
  > while processing commit .
  > rm: cannot remove directory 
`/users/akhaleel/akhaleel/MergeSpace/qemu_0.14_q35/
  > qemu/.git/clone-tmp': Directory not empty
  > 
  > Adnan
  > 
  > 
  > ━
  > From: Isaku Yamahata [mailto:yamah...@valinux.co.jp]
  > To: Hu Tao [mailto:hu...@cn.fujitsu.com], Adnan Khaleel
  > [mailto:ad...@khaleel.us]
  > Cc: qemu-devel@nongnu.org
  > Sent: Wed, 20 Apr 2011 17:46:44 -0500
  > Subject: Re: [Qemu-devel] [PATCH 00/26] q35 chipset support for native 
pci
  > express support
  > 
  > I forgot to changet its HEAD. Now it's fixed.
  > So please change the branch manually or clone the repo again.
  > 
  > On Tue, Apr 19, 2011 at 04:58:32PM +0800, Hu Tao wrote:
  > > On Tue, Apr 19, 2011 at 05:51:27PM +0900, Isaku Yamahata wrote:
  > > > On Tue, Apr 19, 2011 at 04:28:01PM +0800, Hu Tao wrote:
  > > > > On Wed, Mar 16, 2011 at 06:29:11PM +0900, Isaku Yamahata wrote:
  > > > > > This patch series adds basic q35 chipset support for native pci
  > express
  > > > > > support. Some bios related patches are still needed.
  > > > > > For those who want to try it, the following repo is avaiable.
  > > > > > (vgabios doesn't need patches, so use the upstream one)
  > > > > >
  > > > > > git clone 
http://people.valinux.co.jp/~yamahata/qemu/q35/20110316/
  > qemu
  > > > > > git clone 
http://people.valinux.co.jp/~yamahata/qemu/q35/20110316/
  > seabios
  > > > >
  > > > > Hi,
  > > > >
  > > > > When I visit the links, the pages say 'You dont have permissi

[Qemu-devel] adding search to dhcp

[Carl Karsten]

man qemu...
hostname=name   Specifies the client hostname reported by the builtin
DHCP server.

I would like to add a technically similar option: search
It would be sent along the same path that the above hostname parameter takes.

This would allow the guest OS to reference other boxes on my LAN by
just hostname, not FQDN. I have many scripts that rely on this, and so
they break when I try to run them in a qemu vm.


man dhcp-options
   option domain-search domain-list;

 The domain-search option specifies a 'search list' of Domain Names to
 be used by the client to  locate  not-fully-qualified  domain  names.
 The  difference  between  this option and historic use of the domain-
 name option for the same ends is  that  this  option  is  encoded  in
 RFC1035 compressed labels on the wire.  For example:

   option domain-search "example.com", "sales.example.com",
"eng.example.com";

I would expect the syntax to look like this:

qemu -hda 1.qcow2 -net nick -net
user,hostname=qemu,search=example.com,sales.example.com


-- 
Carl K

[Qemu-devel] [PATCH] vfio: Allow sub-ranges to be unmapped

[Alex Williamson]

We're currently very strict in requiring that DMA unmaps are done
using the exact same regions as used for the original mapping.
In a VM environment, we may want to change sub-areas of a previous
mapping without tearing down the entire region.  This might also
also us to support ballooning or memory hotplug in a guest making
use of a vfio device.

Signed-off-by: Alex Williamson 
---

Tom,

I'm a little confused why we have the list of mapped areas attached
to the listener.  If we want to support multiple devices sharing the same
uiommu, don't we instead need to create a separate object hosting this
list with reference counting so multiple vdevs can point to it?  I would
expect that's where the locked_page accounting would be moved to as well.
There also seems to be support that each vdev could have multiple
listeners, each with their own list of dma mapped areas, but we can
only have one udomain per vdev, which just further confuses me.  Can you
hlp me out?  Thanks,

Alex

Note the change of npage to unsigned long below is just to avoid needing
to cast it every time we do a << PAGE_SHIFT to avoid overflow.

 drivers/vfio/vfio_dma.c  |  486 ++
 drivers/vfio/vfio_main.c |8 -
 include/linux/vfio.h |   10 -
 3 files changed, 362 insertions(+), 142 deletions(-)

diff --git a/drivers/vfio/vfio_dma.c b/drivers/vfio/vfio_dma.c
index 4a488b6..bf260e4 100644
--- a/drivers/vfio/vfio_dma.c
+++ b/drivers/vfio/vfio_dma.c
@@ -44,7 +44,7 @@
 
 struct vwork {
struct mm_struct *mm;
-   int npage;
+   unsigned long npage;
struct work_struct work;
 };
 
@@ -62,7 +62,7 @@ static void vfio_lock_acct_bg(struct work_struct *work)
kfree(vwork);
 }
 
-static void vfio_lock_acct(int npage)
+static void vfio_lock_acct(unsigned long npage)
 {
struct vwork *vwork;
struct mm_struct *mm;
@@ -97,27 +97,19 @@ static void vfio_lock_acct(int npage)
 
 /* Unmap DMA region */
 /* dgate must be held */
-static void vfio_dma_unmap(struct vfio_listener *listener,
-   struct dma_map_page *mlp)
+static void vfio_dma_unmap(struct vfio_listener *listener, unsigned long iova,
+  unsigned long npage, struct page **pages, int rdwr)
 {
-   int i;
struct vfio_dev *vdev = listener->vdev;
-   int npage;
-
-   list_del(&mlp->list);
-   for (i = 0; i < mlp->npage; i++)
-   (void) uiommu_unmap(vdev->udomain,
-   mlp->daddr + i * PAGE_SIZE, 0);
-   for (i = 0; i < mlp->npage; i++) {
-   if (mlp->rdwr)
-   SetPageDirty(mlp->pages[i]);
-   put_page(mlp->pages[i]);
+   unsigned long i;
+
+   for (i = 0; i < npage; i++, iova += PAGE_SIZE) {
+   uiommu_unmap(vdev->udomain, iova, 0);
+   if (rdwr)
+   SetPageDirty(pages[i]);
+   put_page(pages[i]);
}
-   vdev->mapcount--;
-   npage = mlp->npage;
-   vdev->locked_pages -= mlp->npage;
-   vfree(mlp->pages);
-   kfree(mlp);
+   vdev->locked_pages -= npage;
vfio_lock_acct(-npage);
 }
 
@@ -130,29 +122,210 @@ void vfio_dma_unmapall(struct vfio_listener *listener)
mutex_lock(&listener->vdev->dgate);
list_for_each_safe(pos, pos2, &listener->dm_list) {
mlp = list_entry(pos, struct dma_map_page, list);
-   vfio_dma_unmap(listener, mlp);
+   vfio_dma_unmap(listener, mlp->daddr, mlp->npage,
+  mlp->pages, mlp->rdwr);
+   list_del(&mlp->list);
+   vfree(mlp->pages);
+   kfree(mlp);
}
mutex_unlock(&listener->vdev->dgate);
 }
 
+/* Map DMA region */
+/* dgate must be held */
+static int vfio_dma_map(struct vfio_listener *listener, unsigned long iova,
+   unsigned long npage, struct page **pages, int rdwr)
+{
+   struct vfio_dev *vdev = listener->vdev;
+   unsigned long i;
+   int ret;
+
+   /* Verify pages are not already mapped */
+   for (i = 0; i < npage; i++)
+   if (uiommu_iova_to_phys(vdev->udomain, iova + i * PAGE_SIZE))
+   return -EBUSY;
+
+   for (i = 0; i < npage; i++, iova += PAGE_SIZE) {
+   ret = uiommu_map(vdev->udomain, iova,
+page_to_phys(pages[i]), 0, rdwr);
+   if (!ret)
+   continue;
+
+   /* Back out mappings on error */
+   for (i--, iova -= PAGE_SIZE; i >= 0; i--, iova -= PAGE_SIZE)
+   uiommu_unmap(vdev->udomain, iova, 0);
+   return ret;
+   }
+   vdev->locked_pages += npage;
+   vfio_lock_acct(npage);
+   return 0;
+}
+
+static struct dma_map_page *vfio_find_dma(struct vfio_listener *listener,
+ dma_addr_t start, size_t size)
+{
+   struct list

Re: [Qemu-devel] Binary translation (of code)

[Tarmo Pikaro]

Message: 3
Date: Sun, 08 May 2011 21:41:15 +0200
From: Llu?s 
To: qemu-devel@nongnu.org
Subject: Re: [Qemu-devel] Binary translation (of code)
Message-ID: <87ei49ugis@ginnungagap.bsc.es>
Content-Type: text/plain; charset=utf-8

Tarmo Pikaro writes:

> Hi !
>> > I think self-modifying?code is kinda rare case - it's made typically for
>> > protection againt?hackers , and?typically on pc side. Nintendo?roms 
probably
>> > don't use this kind of
>> > protection.
>> 
>> It doesn't have to be protection, for example GCC generates
>> trampolines to stack when using nested functions.

> Ok, so apparently we will have some challenges on the way.

> I guess some sort of mutation - qemu + recompiler could be done as well...

> It would be much easier to ship an executable containing both the guest
> application and qemu, so that executing it starts qemu with a
> pre-defined configuration and runs the guest binary.
 
- Binary recompilation would allow faster execution than emulated code
 
- qemu constantly changes (based on amount of patches provided in this
mail list) - in order to keep image+qemu working - simplest way is to convert 
into
executable. Btw - qemu could be used as a bootstrap to image as well.
:-)
 
- And finally modular "emulation" - it would be possible to recompile individual
.dll to another os.
 
Side effects would be:
 
- More difficult to debug - since mapping to original binary image should be 
somehow
maintained - if register dump helps anyhow the developer.
 --
Have a nice day!
Tarmo.

Re: [Qemu-devel] Howto debug boot device not showing up in bios

[Adnan Khaleel]

I meant what is the best way to debug the qemu and seabios interaction, I don't 
necessarily want to inspect instructions but rather just the function calls and 
trace the process so I can compare it for another case that isn't working.


AK
  _  

From: Mulyadi Santosa [mailto:mulyadi.sant...@gmail.com]
To: ad...@khaleel.us
Cc: qemu-devel@nongnu.org
Sent: Tue, 10 May 2011 01:49:31 -0500
Subject: Re: [Qemu-devel] Howto debug boot device not showing up in bios

Hi
  
  On Tue, May 10, 2011 at 05:24, Adnan Khaleel  wrote:
  > Can somebody give me some pointers on what the best way to debug the boot
  > process in Qemu and seabios?
  
  At first, I guess "-s" a.k.a qemu gdb stub of Qemu could help
  you...but uhm, on a second thought, I think it is the qemu itself that
  you need to run under Qemu...
  
  regarding seabios code, i have no idea how to debug that... is it a
  binary blob only?
  
  -- 
  regards,
  
  Mulyadi Santosa
  Freelance Linux trainer and consultant
  
  blog: the-hydra.blogspot.com
  training: mulyaditraining.blogspot.com

Re: [Qemu-devel] Binary translation (of code)

[Richard Henderson]

On 05/10/2011 11:13 AM, Tarmo Pikaro wrote:
>> It would be much easier to ship an executable containing both the guest
>> application and qemu, so that executing it starts qemu with a
>> pre-defined configuration and runs the guest binary.
>  
> - Binary recompilation would allow faster execution than emulated code

Digital Equipment Corp (DEC) did a lot of work with static binary recompilation
in the early 1990's, converting VAX VMS applications to Alpha VMS. It's
quite possible to do if you spend enough time on it, and have a well
defined application environment.

That said, Hewlett Packard (HP) has done a very similar amount of work with
dynamic binary recompilation of PA-RISC HP/UX to IA-64 HP/UX, and achieved
similar results to what DEC achieved.

There has been a *lot* of papers about dynamic recompilation over the last
decade or two.  I believe that the general consensus is that -- with the
addition of dynamic profiling -- dynamic recompilation allows faster 
execution than static recompilation.

A lot of this is stuff that QEMU doesn't do. But the gist is, you add
profiling information to basic blocks as you translate them. This first
compilation pass is very quick and dirty, producing only moderately poor
translated code. As the program runs, a profile is collected that allows
the emulation environment to identify portions of the program that should
be compiled again, with much higher optimization. The thing that allows
this dynamic compilation to produce code that runs faster than static
compilation is that the VM can make simplifying assumptions about how a
portion of the program acts (either discovered from the profile, or a true
guess) and check those assumptions before the translated code is executed.
If the assumptions turn out to be invalid, then the VM can fall back to
the original quick compilation, or re-compile the portion of the program
without the assumptions.

If you're truly interested, a fair portion of these sorts of papers are
written in the context of Java Virtual Machines.  But the techniques apply
equally well to any dynamic compilation process.


r~


P.S: I seem to recall reading that HP had done some testing of their PA-RISC
dynamic recompiler, producing PA-RISC output too.  The recompiled programs
could then run on the same hardware as the original program.  The recompiled
programs ran faster than the originals, due to the techniques described.

Re: [Qemu-devel] Binary translation (of code)

[Lluís]

Tarmo Pikaro writes:
>> It would be much easier to ship an executable containing both the guest
>> application and qemu, so that executing it starts qemu with a
>> pre-defined configuration and runs the guest binary.
>
> - Binary recompilation would allow faster execution than emulated code

QEMU does precisely that, using a JIT. You wouldn't obtain much more
performance by generating a pre-translated binary instead of using
QEMU's JIT. And that's without taking into account the difficulties
associated with the static binary analysis that would be required (as
others have already pointed to you to in previous mails).


> - qemu constantly changes (based on amount of patches provided in this
> mail list) - in order to keep image+qemu working - simplest way is to convert 
> into
> executable. Btw - qemu could be used as a bootstrap to image as well.
> :-)

Maybe you didn't understand what I was saying. Think of it as a
self-extracting executable that contains both a specific qemu binary, as
well as the guest binary you want to execute. When you execute that
bundle, it transparently extracts both components (qemu + the guest
application) and starts qemu to execute that application.

Still, this is a poor approach from the software maintenance point of
view, and falls into the kind of strategies that windows application
developers use (bundle private copies of each library the application is
going to use).


> - And finally modular "emulation" - it would be possible to recompile 
> individual
> .dll to another os.

I suppose you meant another architecture, as recompiling to another OS
makes no sense on the general case (can have a completely different set
of syscalls).

Still, I don't see the point of translating a single library instead of
the whole application (specially when you have access to a library
compiled for you target architecture, or even better access to the
source code to compile that library to whatever target you desire).


> Side effects would be:
>
> - More difficult to debug - since mapping to original binary image should be 
> somehow
> maintained - if register dump helps anyhow the developer.

As others have told you, qemu already has an interface to allow gdb to
debug the guest application.


Lluis

-- 
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth

[Qemu-devel] TCG: AREG0 removal planning

[Blue Swirl]

Hi,

TCG uses a fixed global register (AREG0) to which points to currently
used CPUState, also known as 'env'. Using a fixed register has the
downsides that the register must be reserved by TCG for generated code
and by the compiler for compiling a few critical files (op_helper.c
etc.). The latter also means that any calls to C library may be unsafe
from those files.

Here are my sketches about transition to AREG0-less world.

TCG the generator backend
-AREG0 is used for qemu_ld/st ops for TLB access. It should be
possible for the translators to pass instead a pointer to either
CPUState or directly to the TLB. New qemu_ld/st ops are needed for all
TCG targets.
-TCG temps are stored in CPUState field temp_buf[], accessed via
AREG0. Maybe a regular stack frame should be allocated instead?
-A generator can support translators with and without AREG0 without
performance impact and minor #ifdeffery.

Translators/op helpers
-Op helpers should not use global 'env' but take a CPUState or more
specific pointer instead. The converted helpers should be moved from
op_helper.c to helper.c. As Paul suggested, a new TCG_CALL_TYPE could
be added.
-cpu_env: still needed. Maybe a new TCG temp type should be added?
Special magic temp set up by prologue?
-New qemu_ld/st ops must be used. This breaks the translator when used
with the TCG backends that aren't yet converted, except with heavy
#ifdeffery.
-When a translator is completely AREG0 free, the register can be freed
for normal allocations.
-Performance will degrade until AREG0 usage is eliminated, then
performance should be better than now.

Other TCG execution loop (cpu-exec.c)
-Convert global 'env' use.
-It may be a bit tricky (some #ifdeffery) to support simultaneously
targets that use AREG0 and those that don't.
-Performance may degrade for the QEMU targets that still use AREG0.

Staging
-Ideally generators should be converted first to avoid breakages.
-In practice, breakages and performance regressions will be hard to avoid.

Cleanups
-HELPER_CFLAGS will be eliminated.
-All of QEMU will be plain ordinary C code for easier portability.
-Buggy C libraries (Sparc glibc longjmp destroying %g registers...)
won't be able to mess with global registers, no restrictions about C
library calls from op helpers are needed.
-dyngen_exec.h can be finally eliminated.
-Since qemu_ld/st uses need fixing anyway, the ops can be refactored
as well, for example taking Aurelien's constant patches into account.
-Generic softfloat and other common ops could be added, called
directly without a helper.

History
-In dyngen times, there used to be three global registers, AREG0 to
AREG2 (even more were defined, but not used).
-AREG1 and AREG2 were known as T1 and T2. Dyngen ops used these directly.

Comments? There are a few blank spots too.

Re: [Qemu-devel] TCG: AREG0 removal planning

[Paul Brook]

> TCG uses a fixed global register (AREG0) to which points to currently
> used CPUState, also known as 'env'. Using a fixed register has the
> downsides that the register must be reserved by TCG for generated code
> and by the compiler for compiling a few critical files (op_helper.c
> etc.). The latter also means that any calls to C library may be unsafe
> from those files.
> 
> Here are my sketches about transition to AREG0-less world.
> ...
> Translators/op helpers
> ...
> Comments? There are a few blank spots too.

I think a useful, and incremental goal is elimination of global cpu_env state 
in C code (i.e eliminate HELPER_CFLAGS and dyngen-exec.h).
We already have much of the infrastructure for this - op_helper v.s. helper.c 
and code_gen_prologue for transition in/out of "generated code" state.

In practice generated code probably accesses CPUState often enough that a 
dedicated register isn't a bad idea.  My guess is that eliminating it from C 
code gets us almost all of the useful benefit.  Removing it from the code 
generator (i.e. TCG_AREG0) may be more pain that it's worth.  For changes to 
the TCG side we want to consider how we can provide useful aliasing 
information, rather than a naive replacement of TCG_AREG0 with a variable.

Paul

Re: [Qemu-devel] TCG: AREG0 removal planning

[Stefan Weil]

Am 10.05.2011 22:54, schrieb Blue Swirl:

Hi,

TCG uses a fixed global register (AREG0) to which points to currently
used CPUState, also known as 'env'. Using a fixed register has the
downsides that the register must be reserved by TCG for generated code
and by the compiler for compiling a few critical files (op_helper.c
etc.). The latter also means that any calls to C library may be unsafe
from those files.

Here are my sketches about transition to AREG0-less world.

TCG the generator backend
-AREG0 is used for qemu_ld/st ops for TLB access. It should be
possible for the translators to pass instead a pointer to either
CPUState or directly to the TLB. New qemu_ld/st ops are needed for all
TCG targets.
-TCG temps are stored in CPUState field temp_buf[], accessed via
AREG0. Maybe a regular stack frame should be allocated instead?
-A generator can support translators with and without AREG0 without
performance impact and minor #ifdeffery.

Translators/op helpers
-Op helpers should not use global 'env' but take a CPUState or more
specific pointer instead. The converted helpers should be moved from
op_helper.c to helper.c. As Paul suggested, a new TCG_CALL_TYPE could
be added.
-cpu_env: still needed. Maybe a new TCG temp type should be added?
Special magic temp set up by prologue?
-New qemu_ld/st ops must be used. This breaks the translator when used
with the TCG backends that aren't yet converted, except with heavy
#ifdeffery.
-When a translator is completely AREG0 free, the register can be freed
for normal allocations.
-Performance will degrade until AREG0 usage is eliminated, then
performance should be better than now.

Other TCG execution loop (cpu-exec.c)
-Convert global 'env' use.
-It may be a bit tricky (some #ifdeffery) to support simultaneously
targets that use AREG0 and those that don't.
-Performance may degrade for the QEMU targets that still use AREG0.

Staging
-Ideally generators should be converted first to avoid breakages.
-In practice, breakages and performance regressions will be hard to avoid.

Cleanups
-HELPER_CFLAGS will be eliminated.
-All of QEMU will be plain ordinary C code for easier portability.
-Buggy C libraries (Sparc glibc longjmp destroying %g registers...)
won't be able to mess with global registers, no restrictions about C
library calls from op helpers are needed.
-dyngen_exec.h can be finally eliminated.
-Since qemu_ld/st uses need fixing anyway, the ops can be refactored
as well, for example taking Aurelien's constant patches into account.
-Generic softfloat and other common ops could be added, called
directly without a helper.

History
-In dyngen times, there used to be three global registers, AREG0 to
AREG2 (even more were defined, but not used).
-AREG1 and AREG2 were known as T1 and T2. Dyngen ops used these directly.

Comments? There are a few blank spots too.


Just for information:

TCI (the TCG interpreter) uses a global variable (no register,
because it is designed to run on any architecture).

Code extracts:

dyngen-exec.h:
#if defined(AREG0)
# define DECLARE_QEMU_ENV(s) register struct s *env asm(AREG0)
#else
# define DECLARE_QEMU_ENV(s) extern struct s *env
#endif

target-i386/exec.h (similar for other targets):
DECLARE_QEMU_ENV(CPUX86State);

Your plan might allow co-existence of the normal TCG and TCI in
the same binary, so users could select TCI via a command line
switch (there are a few use cases where TCI has advantages).

It is also a small step towards a multicore emulation which makes
full use of real multicore cpus.

I have some regression tests here (mainly debian linux guests
with different architectures, but also an unspeakable host os
which some people don't like) and will certainly run these tests
when required.

And yes, you will discover more blank spots when you do the work.

Regards,
Stefan

Re: [Qemu-devel] [PATCH 00/35] Alpha system emulation, v4

[Paul Brook]

> Since virtio devices intentionally access memory directly, we
> are not actually dependant on the iommu patches in order to
> make progress.  Merely fixing the PCI interrupt setup was
> enough to get the virtio-pci interface working.

There should be nothing special about virtio-pci. These devices should access 
memory in exactly the same way as any other PCI device. Bypassing the normal 
PCI iommu is IMO a serious bug.

Paul

Re: [Qemu-devel] TCG: AREG0 removal planning

[Richard Henderson]

On 05/10/2011 01:54 PM, Blue Swirl wrote:
> TCG the generator backend
> -AREG0 is used for qemu_ld/st ops for TLB access. It should be
> possible for the translators to pass instead a pointer to either
> CPUState or directly to the TLB.

I believe that AREG0 should continue to be present in the generated
code.  There are simply too many references to it throughout the
translated code for allocating this dynamically to be a win.

What should change, however, is the removal of AREG0 outside the
generated code.  The cpu-state pointer should be passed as a regular
parameter wherever it is required.  This includes tcg_qemu_tb_exec,
which means that the generated prologue would change, setting up
AREG0 in the process.

> New qemu_ld/st ops are needed for all TCG targets.

Yes, qemu_ld/st would have to change to accommodate the new parameter
being passed.

While we're at it, let us change things a bit further to allow guest
byte-swap load/store insns to be implemented more efficiently.  For
instance, currently a sparc load_asr (little-endian), as emulated on
an x86 host, does the byte swap twice.

There is, currently, a const int parameter to qemu_ld/st that encodes
the size of the load.  Almost all TCG backends behind the scenes 
extend this parameter with a bit to indicate byte swap needed.  Let us
formalize this, and allow this to be set in the original TCG op, with
appropriate new inlines in tcg-op.h to access it from the translators.

We can also make things easier for the backends by allowing them
to declare that they do or do not have byte swap load/store insns.
If the such are not available, a separate bswap opcode is emitted
right from tcg_gen_qemu_st32 et al.

This would allow a nice cleanup for i386, which currently has a small
register allocation problem in the store path, what with needing to
not clobber the input register while byte swapping.  (This problem is
solved by restricting the set of input registers for qemu_ld/st.)

All this does require the slow path to be changed to accommodate this.
In particular, if byte-swap memory ops are available, we need slow
path functions that also byte swap.  Indeed, I'd expect them to use
the byte-swap memory ops themselves.  Further, if byte-swap memory
ops are not available, the slow path should always return memory in
the host byte order, because a separate bswap operation will be done
on behalf of the fast path.

> -TCG temps are stored in CPUState field temp_buf[], accessed via
> AREG0. Maybe a regular stack frame should be allocated instead?

Probably.  Most of the backends manage a stack frame anyway, to
handle registers saved in the prologue.  All that would be needed
is a define from TCG to tell the backends how much memory is required,
and some value passed from the backends to tell TCG what the offset
of that area is from the stack pointer.


r~

Re: [Qemu-devel] [PATCH 00/35] Alpha system emulation, v4

[Richard Henderson]

On 05/10/2011 02:33 PM, Paul Brook wrote:
> There should be nothing special about virtio-pci. These devices should access 
> memory in exactly the same way as any other PCI device. Bypassing the normal 
> PCI iommu is IMO a serious bug.

I suspect that the argument is that virtio is implemented on several
busses, and ignoring any bus-specific stuff means that there's less
need for any code differences.

It doesn't bother me either way, so long as I know what to expect.


r~

Re: [Qemu-devel] [PATCH 00/35] Alpha system emulation, v4

[Paul Brook]

> On 05/10/2011 02:33 PM, Paul Brook wrote:
> > There should be nothing special about virtio-pci. These devices should
> > access memory in exactly the same way as any other PCI device. Bypassing
> > the normal PCI iommu is IMO a serious bug.
> 
> I suspect that the argument is that virtio is implemented on several
> busses, and ignoring any bus-specific stuff means that there's less
> need for any code differences.

It also has some horrifically bad implications. Not least of which is the 
inability to sanely handle nested virtualization.

IMO the whole point of virtio-pci is that it avoids magical weirdness for 
virtio devices devices.  If you're going to start adding nonstandard 
exceptions for virtio devices than there seem little point using PCI in the 
first place.  It's likely to cause more problems than it solves for the guest 
drivers.

Paul

Re: [Qemu-devel] TCG: AREG0 removal planning

[Paul Brook]

> While we're at it, let us change things a bit further to allow guest
> byte-swap load/store insns to be implemented more efficiently.  For
> instance, currently a sparc load_asr (little-endian), as emulated on
> an x86 host, does the byte swap twice.

FWIW this also ends up interacting with the device and bus models. This is 
partially implemented by the endian parameter of cpu_register_io_memory et. 
al.  This may also be a runtime property, either part of the CPU state (e.g. 
ARM where instruction and data accesses may have different endianness), or 
even a per-page TLB attribute (PPC?).

Paul

[Qemu-devel] [PATCH] Add AACI audio playback support to the ARM Versatile/PB platform

[Mathieu Sonet]

The PL041 driver provides an interface to an ACLink bus.
The LM4549 driver emulates a DAC connected on the ACLink bus.
Only audio playback is implemented.

Versatile/PB test build:
linux-2.6.38.5
buildroot-2010.11
alsa-lib-1.0.22
alsa-utils-1.0.22
mpg123-0.66

Qemu host: Ubuntu 10.04 in Vmware/OS X

Playback tested successfully with aplay and mpg123.

Signed-off-by: Mathieu Sonet 
---
 Makefile.target  |1 +
 hw/aclink.c  |  121 +++
 hw/aclink.h  |   63 
 hw/lm4549.c  |  368 +
 hw/pl041.c   |  436 ++
 hw/pl041.h   |  126 
 hw/pl041.hx  |   62 
 hw/versatilepb.c |6 +
 8 files changed, 1183 insertions(+), 0 deletions(-)
 create mode 100644 hw/aclink.c
 create mode 100644 hw/aclink.h
 create mode 100644 hw/lm4549.c
 create mode 100644 hw/pl041.c
 create mode 100644 hw/pl041.h
 create mode 100644 hw/pl041.hx

diff --git a/Makefile.target b/Makefile.target
index 21f864a..cdd7b40 100644
--- a/Makefile.target
+++ b/Makefile.target
@@ -354,6 +354,7 @@ obj-arm-y += syborg_virtio.o
 obj-arm-y += vexpress.o
 obj-arm-y += strongarm.o
 obj-arm-y += collie.o
+obj-arm-y += pl041.o aclink.o lm4549.o

 obj-sh4-y = shix.o r2d.o sh7750.o sh7750_regnames.o tc58128.o
 obj-sh4-y += sh_timer.o sh_serial.o sh_intc.o sh_pci.o sm501.o
diff --git a/hw/aclink.c b/hw/aclink.c
new file mode 100644
index 000..c335f60
--- /dev/null
+++ b/hw/aclink.c
@@ -0,0 +1,121 @@
+/*
+ * ACLink Interface
+ *
+ * Copyright (c) 2011
+ * Written by Mathieu Sonet - www.elasticsheep.com
+ *
+ * This code is licenced under the GPL.
+ *
+ * *
+ *
+ * This file defines the ACLink bus interface to exchange data
+ * between an host and a codec.
+ *
+ */
+
+#include "aclink.h"
+
+/*** Types ***/
+
+struct ACLinkBus {
+BusState qbus;
+ACLinkControllerInfo *controller_info;
+uint32_t bitclk;
+};
+
+struct BusInfo aclink_bus_info = {
+.name = "aclink",
+.size = sizeof(ACLinkBus),
+};
+
+/*** Functions ***/
+
+ACLinkBus *aclink_create_bus(DeviceState *parent, const char *name)
+{
+BusState *bus;
+bus = qbus_create(&aclink_bus_info, parent, name);
+return FROM_QBUS(ACLinkBus, bus);
+}
+
+void aclink_set_controller_info(ACLinkBus *bus, ACLinkControllerInfo *info)
+{
+bus->controller_info = info;
+}
+
+static int aclink_device_init(DeviceState *dev, DeviceInfo *base_info)
+{
+ACLinkDeviceInfo *info = container_of(base_info, ACLinkDeviceInfo, qdev);
+ACLinkDevice *s = ACLINK_DEVICE_FROM_QDEV(dev);
+ACLinkBus *bus;
+
+bus = FROM_QBUS(ACLinkBus, qdev_get_parent_bus(dev));
+if (QLIST_FIRST(&bus->qbus.children) != dev
+|| QLIST_NEXT(dev, sibling) != NULL) {
+hw_error("Too many devices on the ACLINK bus");
+}
+
+s->info = info;
+return info->init(s);
+}
+
+void aclink_register_device(ACLinkDeviceInfo *info)
+{
+assert(info->qdev.size >= sizeof(ACLinkDevice));
+info->qdev.init = aclink_device_init;
+info->qdev.bus_info = &aclink_bus_info;
+qdev_register(&info->qdev);
+}
+
+DeviceState *aclink_create_device(ACLinkBus *bus, const char *name)
+{
+DeviceState *dev;
+dev = qdev_create(&bus->qbus, name);
+qdev_init_nofail(dev);
+return dev;
+}
+
+static ACLinkDevice *aclink_get_device(ACLinkBus *bus)
+{
+DeviceState *dev = QLIST_FIRST(&bus->qbus.children);
+if (!dev) {
+return NULL;
+}
+return ACLINK_DEVICE_FROM_QDEV(dev);
+}
+
+void aclink_bitclk_enable(ACLinkDevice *dev, uint32_t on)
+{
+ACLinkBus *bus = FROM_QBUS(ACLinkBus, qdev_get_parent_bus(&dev->qdev));
+uint32_t has_changed;
+
+on = (on > 0) ? 1 : 0;
+has_changed = (bus->bitclk != on) ? 1 : 0;
+
+bus->bitclk = on;
+if (has_changed) {
+bus->controller_info->bitclk_state_changed(bus->qbus.parent);
+}
+}
+
+uint32_t aclink_bitclk_is_enabled(ACLinkBus *bus)
+{
+return bus->bitclk;
+}
+
+void aclink_sdataout_slot12(ACLinkBus *bus, uint32_t slot1, uint32_t slot2)
+{
+ACLinkDevice *device = aclink_get_device(bus);
+device->info->sdataout_slot12(device, slot1, slot2);
+}
+
+void aclink_sdataout_slot34(ACLinkBus *bus, uint32_t slot3, uint32_t slot4)
+{
+ACLinkDevice *device = aclink_get_device(bus);
+device->info->sdataout_slot34(device, slot3, slot4);
+}
+
+void aclink_sdatain_slot12(ACLinkDevice *dev, uint32_t slot1, uint32_t slot2)
+{
+ACLinkBus *bus = FROM_QBUS(ACLinkBus, qdev_get_parent_bus(&dev->qdev));
+bus->controller_info->sdatain_slot12(bus->qbus.parent, slot1, slot2);
+}
diff --git a/hw/aclink.h b/hw/aclink.h
new file mode 100644
index 000..d360d4b
--- /dev/null
+++ b/hw/aclink.h
@@ -0,0 +1,63 @@
+/*
+ * ACLink Interface
+ *
+ * Copyright (c) 2011
+ * Written by Mathieu Sonet - www.elasticsheep.com
+ *
+ * This code is licenced under the GPL.
+ *
+ * 

Re: [Qemu-devel] [PATCH] Add AACI audio playback support to the ARM Versatile/PB platform

[malc]

On Tue, 10 May 2011, Mathieu Sonet wrote:

> The PL041 driver provides an interface to an ACLink bus.
> The LM4549 driver emulates a DAC connected on the ACLink bus.
> Only audio playback is implemented.
> 
> Versatile/PB test build:
> linux-2.6.38.5
> buildroot-2010.11
> alsa-lib-1.0.22
> alsa-utils-1.0.22
> mpg123-0.66
> 
> Qemu host: Ubuntu 10.04 in Vmware/OS X
> 
> Playback tested successfully with aplay and mpg123.
> 
> Signed-off-by: Mathieu Sonet 

Looks fine to me.

-- 
mailto:av1...@comtv.ru

Re: [Qemu-devel] Allow ARMv7M to be started without a kernel

[Rob Landley]

On 05/10/2011 12:13 AM, Alexander Graf wrote:
> 
> On 10.05.2011, at 06:58, Rob Landley wrote:
> 
>> On 05/09/2011 09:11 AM, Alexander Graf wrote:
 C) requires more research, because I have to make sure the
 entry point is either doing the 16->32 (or 64) bit startup
 dance or that it's being launched in the right mode (which the
 bios isn't doing), but vmlinux doesn't need to be decompressed
 and copied so it's just making sure we're expecting the right
 layer of stuff.  Which means reading through arch/x86/boot to
 see what the vmlinux->bzImage packaging is adding, I suppose.
>>> 
>>> The issue is that this is not how it works on real hardware.
>> 
>> Real hardware doesn't have the -kernel option.  On x86 that option
>> is currently patching the rdev area after loading the blob into
>> DRAM.
>> 
>>> Grub won't just load a vmlinux file and boot it.
>> 
>> The -kernel option doesn't load grub, it loads a _kernel_.  -kernel
>> is a linux-specific bootloader.  It doesn't chainload other
>> bootloaders.
> 
> My point is that -kernel is on the same level as grub. It's a
> bootloader - just (mostly) implemented in host mode, not guest mode.
> I'm not objecting to your idea. Just saying that what you're doing
> isn't streamlined in Linux atm, so you might hit barriers that people
> didn't expect :).

Oh sure.  That's why I haven't posted the patch yet.

I read through enough to assemble a very rough todo list (call
load_elf(), figure out if the kernel can run from the read-only ROM blob
or if I need to copy it to normal DRAM like some targets do, figure out
where the structure rdev used to patch lives in the blob the ELF loaded
so we can set the right field in it to specify where the command line
lives, figure out what we need to add to said command line...)

But it's not in the top 5 of my todo list today...

>> By the way, I just did "qemu -kernel qemu-img" an the emulated vga 
>> screen was very colorful.  Feeding it qemu-io gave a different
>> pattern of vga text mode garbage.  It's happily loading an elf file
>> _now_, and spinning its little CPU on the resulting garbage...
> 
> IIRC -kernel loads ELF images as raw blobs atm. Just try to implement
> it and you'll see what I mean, seriously :).

Which is why -kernel needs to fix up the result (if nothing else, to
feed it a command line).

> Basically, -kernel with an OpenBSD ELF kernel would be different.

My lack of caring about OpenBSD cannot be expressed in words.  Does the
current -kernel option support OpenBSD kernels?  If so, I wouldn't be
introducing any regressions.

> You somehow need to be able to find out the boot protocol.

No, I don't.  I need to ignore OpenBSD with the force of a thousand DMV
agents on break.

"Your new thing would continue not to do something the current -kernel
option doesn't support".  Yes.  Yes it would.  I don't see how this is
considered a valid objection.

> Either it's
> defined by architecture or by random OS conventions. If you can
> somehow make sure that a kernel really is Linux and nothing else,
> speaking some random Linux boot protocol with it is just fine!

The current -kernel option will happily load a TEXT FILE.  The result
does not boot, but it makes pretty pictures in the VGA screen.

Feeding -kernel an OpenBSD kernel is only slightly less useful than
feeding it a text file, and if you want multiboot you know where to find
it.  (There's code for that in qemu now, it's just not at all
interesting to me.)

>> Yes, -kernel has to do setup work before starting the kernel.  This
>> is true now, it remains true if the kernel is in a different file
>> format. -kernel is a linux bootloader, which is not the same thing
>> as the -bios option.
> 
> I disagree. -kernel is a _kernel_ bootloader.

A linux kernel bootloader.  It's 100% linux-specific.  Always has been.

Feel free to provide a counter-example.  Show me how to boot an OpenBSD
kernel with "qemu-system-x86_64 -kernel".

> There's more out there than just Linux :).

The current code disagrees with you.  This project's been around for a
decade and nobody's ever bothered to add BSD support to -kernel, and I
won't either.

If you'd be more comfortable if the command line option was renamed
"-linux", feel free to submit a patch, but you're currently arguing
against the existing code, not against the patch I'm writing.

Rob