Re: [PATCH] arm64, numa: Add cpu_to_node() implementation.

2016-09-19 Thread Ganapatrao Kulkarni
606821) > [2.603464] ---[ end trace 58c0cd36b88802bc ]--- > [2.608138] Kernel panic - not syncing: Fatal exception > > Fix by supplying a cpu_to_node() implementation that returns correct > node mappings. > > Cc: # 4.7.x- > Signed-off-by: David Daney > Acked

[PATCH v5 0/4] Add support for ThunderX2 pmu events using json files

2017-08-16 Thread Ganapatrao Kulkarni
. - Rebased to 4.12-rc1 v2: - Updated as per Mark Rutland's suggestions. - Added provision for get_cpuid_str to get cpu id string from associated cpus of pmu core device. v1: Initial patchset. Ganapatrao Kulkarni (4): perf utils: passing pmu as a parameter to function get_cpuid_str

[PATCH v5 2/4] perf tools arm64: Add support for get_cpuid_str function.

2017-08-16 Thread Ganapatrao Kulkarni
function get_cpuid_str returns MIDR string of the first online cpu from the range of cpus associated with the pmu core device. Signed-off-by: Ganapatrao Kulkarni --- tools/perf/arch/arm64/util/Build| 1 + tools/perf/arch/arm64/util/header.c | 61 + 2

[PATCH v5 4/4] perf vendor events arm64: Add ThunderX2 implementation defined pmu core events

2017-08-16 Thread Ganapatrao Kulkarni
This is not a full event list, but a short list of useful events. Signed-off-by: Ganapatrao Kulkarni --- tools/perf/pmu-events/arch/arm64/mapfile.csv | 15 ++ .../arm64/thunderx2/implementation-defined.json| 62 ++ 2 files changed, 77 insertions(+) create mode

[PATCH v5 3/4] perf utils: Add helper function is_pmu_core to detect PMU CORE devices

2017-08-16 Thread Ganapatrao Kulkarni
On some platforms, PMU core devices sysfs name is not cpu. Adding function is_pmu_core to detect as core device using core device specific hints in sysfs. For arm64 platforms, all core devices have file "cpus" in sysfs. Tested-by: Shaokun Zhang Signed-off-by: Ganapatrao Kulkarni

[PATCH v5 1/4] perf utils: passing pmu as a parameter to function get_cpuid_str

2017-08-16 Thread Ganapatrao Kulkarni
cpuid string will not be same on all CPUs on heterogeneous platforms like ARM's big.LITTLE, adding provision(using pmu->cpus) to find cpuid string from associated CPUs of PMU CORE device. Signed-off-by: Ganapatrao Kulkarni --- tools/perf/arch/powerpc/util/header.c | 2 +- tools/perf/

Re: [PATCH v3 1/2] acpica: iort: Update SMMUv3 header for proximity domain mapping

2017-06-19 Thread Ganapatrao Kulkarni
On Mon, Jun 19, 2017 at 11:24 PM, Robin Murphy wrote: > On 08/06/17 05:44, Ganapatrao Kulkarni wrote: >> ARM IORT specification (rev. C) has added two new fields to define >> proximity domain for the SMMUv3 node in the IORT table. >> >> Proximity Domain Valid: >

[PATCH v2 0/2] acpi, gicv3-its, numa: Adding numa node mapping for ITS

2017-06-20 Thread Ganapatrao Kulkarni
ACPI 6.2 have added SRAT subtable to define proximity domain for ITS devices. This patchset updates acpi header file from acpica repo and adds numa node mapping for ITS devices. v2: - Incorporated comments from Lorenzo Pieralisi and Marc Zyngier. v1: first patch Ganapatrao Kulkarni (2

[PATCH v2 2/2] acpi, gicv3-its, numa: Adding numa node mapping for gic-its units

2017-06-20 Thread Ganapatrao Kulkarni
Add code to parse SRAT ITS Affinity sub table as defined in ACPI 6.2 Later in per device probe, ITS devices are mapped to numa node using ITS id to proximity domain mapping. Signed-off-by: Ganapatrao Kulkarni --- drivers/irqchip/irq-gic-v3-its.c | 76 +++- 1

[PATCH v2 1/2] ACPICA: ACPI 6.2: Add support for new SRAT subtable

2017-06-20 Thread Ganapatrao Kulkarni
Add GIC ITS Affinity (ACPI 6.2) subtable to SRAT table. ACPICA commit 5bc67f63918da249bfe279ee461d152bb3e6f55b Link: https://github.com/acpica/acpica/commit/5bc67f6 Signed-off-by: Ganapatrao Kulkarni --- include/acpi/actbl1.h | 12 +++- 1 file changed, 11 insertions(+), 1 deletion

Re: [PATCH v2 2/2] acpi, gicv3-its, numa: Adding numa node mapping for gic-its units

2017-06-20 Thread Ganapatrao Kulkarni
Hi Lorenzo, On Tue, Jun 20, 2017 at 8:47 PM, Lorenzo Pieralisi wrote: > Hi Ganapatrao, > > On Tue, Jun 20, 2017 at 12:37:17PM +0530, Ganapatrao Kulkarni wrote: >> Add code to parse SRAT ITS Affinity sub table as defined in ACPI 6.2 >> Later in per device probe, ITS devices

[PATCH v3 2/2] acpi, gicv3-its, numa: Adding numa node mapping for gic-its units

2017-06-20 Thread Ganapatrao Kulkarni
Add code to parse SRAT ITS Affinity sub table as defined in ACPI 6.2 Later in per device probe, ITS devices are mapped to numa node using ITS id to proximity domain mapping. Signed-off-by: Ganapatrao Kulkarni --- drivers/irqchip/irq-gic-v3-its.c | 80 +++- 1

[PATCH v3 1/2] ACPICA: ACPI 6.2: Add support for new SRAT subtable

2017-06-20 Thread Ganapatrao Kulkarni
Add GIC ITS Affinity (ACPI 6.2) subtable to SRAT table. ACPICA commit 5bc67f63918da249bfe279ee461d152bb3e6f55b Link: https://github.com/acpica/acpica/commit/5bc67f6 Signed-off-by: Ganapatrao Kulkarni --- include/acpi/actbl1.h | 12 +++- 1 file changed, 11 insertions(+), 1 deletion

[PATCH v3 0/2] acpi, gicv3-its, numa: Adding numa node mapping for ITS

2017-06-20 Thread Ganapatrao Kulkarni
and Marc Zyngier. v1: first patch Ganapatrao Kulkarni (2): ACPICA: ACPI 6.2: Add support for new SRAT subtable acpi, gicv3-its, numa: Adding numa node mapping for gic-its units drivers/irqchip/irq-gic-v3-its.c | 80 +++- include/acpi/actbl1.h

Re: [PATCH v3 2/2] acpi, gicv3-its, numa: Adding numa node mapping for gic-its units

2017-06-21 Thread Ganapatrao Kulkarni
On Wed, Jun 21, 2017 at 2:28 PM, Marc Zyngier wrote: > On 21/06/17 07:15, Ganapatrao Kulkarni wrote: >> Add code to parse SRAT ITS Affinity sub table as defined in ACPI 6.2 >> Later in per device probe, ITS devices are mapped to >> numa node using ITS id to proximity domain

Re: [PATCH v3 2/2] acpi, gicv3-its, numa: Adding numa node mapping for gic-its units

2017-06-21 Thread Ganapatrao Kulkarni
On Wed, Jun 21, 2017 at 2:58 PM, Lorenzo Pieralisi wrote: > On Wed, Jun 21, 2017 at 11:45:43AM +0530, Ganapatrao Kulkarni wrote: >> Add code to parse SRAT ITS Affinity sub table as defined in ACPI 6.2 >> Later in per device probe, ITS devices are mapped to >> numa node usin

[PATCH v4 0/2] acpi, gicv3-its, numa: Adding numa node mapping for ITS

2017-06-21 Thread Ganapatrao Kulkarni
: - Incorporated comments from Lorenzo Pieralisi and Marc Zyngier. v1: first patch Ganapatrao Kulkarni (2): ACPICA: ACPI 6.2: Add support for new SRAT subtable acpi, gicv3-its, numa: Adding numa node mapping for gic-its units drivers/irqchip/irq-gic-v3-its.c | 75

[PATCH v4 2/2] acpi, gicv3-its, numa: Adding numa node mapping for gic-its units

2017-06-21 Thread Ganapatrao Kulkarni
Add code to parse SRAT ITS Affinity sub table as defined in ACPI 6.2. Later in per device probe, ITS devices are mapped to numa node using ITS Id to proximity domain mapping. Signed-off-by: Ganapatrao Kulkarni --- drivers/irqchip/irq-gic-v3-its.c | 75 +++- 1

[PATCH v4 1/2] ACPICA: ACPI 6.2: Add support for new SRAT subtable

2017-06-21 Thread Ganapatrao Kulkarni
Add GIC ITS Affinity (ACPI 6.2) subtable to SRAT table. ACPICA commit 5bc67f63918da249bfe279ee461d152bb3e6f55b Link: https://github.com/acpica/acpica/commit/5bc67f6 Signed-off-by: Ganapatrao Kulkarni --- include/acpi/actbl1.h | 12 +++- 1 file changed, 11 insertions(+), 1 deletion

[PATCH v9 0/5] Add support for ThunderX2 pmu events using json files

2017-10-16 Thread Ganapatrao Kulkarni
pu id string from associated cpus of pmu core device. v1: Initial patchset. Ganapatrao Kulkarni (5): perf utils: passing pmu as a parameter to function get_cpuid_str perf tools arm64: Add support for get_cpuid_str function. perf utils: use pmu->is_uncore to detect PMU CORE/UNCORE device

[PATCH v9 1/5] perf utils: passing pmu as a parameter to function get_cpuid_str

2017-10-16 Thread Ganapatrao Kulkarni
cpuid string will not be same on all CPUs on heterogeneous platforms like ARM's big.LITTLE, adding provision(using pmu->cpus) to find cpuid string from associated CPUs of PMU CORE device. also optimise arguments to function pmu_add_cpu_aliases. Acked-by: Will Deacon Signed-off-by: Ga

[PATCH v9 4/5] perf vendor events arm64: Add ThunderX2 implementation defined pmu core events

2017-10-16 Thread Ganapatrao Kulkarni
This is not a full event list, but a short list of useful events. Acked-by: Will Deacon Signed-off-by: Ganapatrao Kulkarni --- .../arch/arm64/cavium/thunderx2-imp-def.json | 62 ++ tools/perf/pmu-events/arch/arm64/mapfile.csv | 15 ++ 2 files changed, 77

[PATCH v9 2/5] perf tools arm64: Add support for get_cpuid_str function.

2017-10-16 Thread Ganapatrao Kulkarni
function get_cpuid_str returns MIDR string of the first online cpu from the range of cpus associated with the PMU CORE device. Acked-by: Will Deacon Signed-off-by: Ganapatrao Kulkarni --- tools/perf/arch/arm64/util/Build| 1 + tools/perf/arch/arm64/util/header.c | 65

[PATCH v9 5/5] perf utils: add check for valid cpuid in function perf_pmu__find_map

2017-10-16 Thread Ganapatrao Kulkarni
On some platforms(arm/arm64) which uses cpus map to get corresponding cpuid string, cpuid can be NULL for PMUs other than CORE PMUs. Adding check for NULL cpuid in function perf_pmu__find_map to avoid segmentation fault. Signed-off-by: Ganapatrao Kulkarni --- tools/perf/util/pmu.c | 6 ++ 1

[PATCH v9 3/5] perf utils: use pmu->is_uncore to detect PMU UNCORE devices

2017-10-16 Thread Ganapatrao Kulkarni
t PMU") has introduced pmu->is_uncore, which is set to PMU UNCORE devices only. Adding changes to use pmu->is_uncore to identify UNCORE devices. Acked-by: Will Deacon Tested-by: Shaokun Zhang Signed-off-by: Ganapatrao Kulkarni --- tools/perf/util/pmu.c | 11 +++ 1 file chang

Re: [RESEND PATCH 0/2] Add ThunderX2 SoC Performance Monitoring Unit driver

2017-05-28 Thread Ganapatrao Kulkarni
Hi Mark, Will, can you please review this patchset? On Wed, May 17, 2017 at 12:30 PM, Ganapatrao Kulkarni wrote: > This adds PMU driver for Cavium's ThunderX2 SoC UNCORE devices. > The SoC has PMU support in its L3 cache controller (L3C) and in the > DDR4 Memory Controller (DMC).

Re: [PATCH v3 0/4] Add support for ThunderX2 pmu events using json files

2017-05-28 Thread Ganapatrao Kulkarni
Any further review comments on this patch series? can it go in 4.13? On Tue, May 16, 2017 at 2:03 PM, Ganapatrao Kulkarni wrote: > Extending json/jevent framework for parsing arm64 event files. > Adding jevents for ThunderX2 implementation defined PMU events. > > v3: >- Add

Re: [PATCH] perf vendor events arm64: Add hip08 implementation defined PMU core events

2017-10-25 Thread Ganapatrao Kulkarni
Hi John, On Thu, Oct 19, 2017 at 3:59 PM, John Garry wrote: >>> >>> #Family-model,Version,Filename,EventType >>> 0x420f5160,v1,cavium,core >>> 0x420f5161,v1,cavium,core >> >> >> certainly, there is Part number(PartNum, bits [15:4] ) change from >> thunderx2 to thunderx3. >> thunde

Re: [PATCH 3/4] iommu/arm-smmu-v3: Use NUMA memory allocations for stream tables and comamnd queues

2017-11-06 Thread Ganapatrao Kulkarni
On Wed, Oct 18, 2017 at 7:06 PM, Robin Murphy wrote: > On 04/10/17 14:53, Ganapatrao Kulkarni wrote: >> Hi Robin, >> >> >> On Thu, Sep 21, 2017 at 5:28 PM, Robin Murphy wrote: >>> [+Christoph and Marek] >>> >>> On 21/09/17 09:59, Ganapatrao

Re: [PATCH] perf vendor events arm64: Add hip08 implementation defined PMU core events

2017-10-18 Thread Ganapatrao Kulkarni
/arm-kernel/msg611895.html >>>>> >>>>> Signed-off-by: Shaokun Zhang >>>>> Cc: Peter Zijlstra >>>>> Cc: Ingo Molnar >>>>> Cc: Arnaldo Carvalho de Melo >>>>> Cc: Alexander Shishkin >>>>> Cc: Will

Re: [PATCH v9 0/5] Add support for ThunderX2 pmu events using json files

2017-10-18 Thread Ganapatrao Kulkarni
Hi Arnaldo, Will, can this be queued to next? On Tue, Oct 17, 2017 at 12:02 AM, Ganapatrao Kulkarni wrote: > Extending json/jevent framework for parsing arm64 event files. > Adding jevents for ThunderX2 implementation defined PMU events. > > v9: >- Rebased to [3] and resol

Re: [PATCH 01/11] ARM64 / PCI: introduce struct pci_controller for ACPI

2015-09-06 Thread Ganapatrao Kulkarni
Hi Hanjun, On Wed, May 27, 2015 at 1:51 PM, Hanjun Guo wrote: > Hi Liviu, > > On 2015年05月27日 01:20, Jiang Liu wrote: >> >> On 2015/5/27 0:58, Liviu Dudau wrote: >>> >>> On Tue, May 26, 2015 at 01:49:14PM +0100, Hanjun Guo wrote: ARM64 ACPI based PCI host bridge init needs a arch depende

[PATCH] perf vendor events arm64: Update ThunderX2 implementation defined pmu core events

2018-07-31 Thread Ganapatrao Kulkarni
Signed-off-by: Ganapatrao Kulkarni --- .../arch/arm64/cavium/thunderx2/core-imp-def.json | 87 +- 1 file changed, 84 insertions(+), 3 deletions(-) diff --git a/tools/perf/pmu-events/arch/arm64/cavium/thunderx2/core-imp-def.json b/tools/perf/pmu-events/arch/arm64/cavium

Re: [PATCH] perf vendor events arm64: Update ThunderX2 implementation defined pmu core events

2018-07-31 Thread Ganapatrao Kulkarni
Hi Arnaldo, On Tue, Jul 31, 2018 at 7:58 PM, Arnaldo Carvalho de Melo wrote: > Em Tue, Jul 31, 2018 at 03:32:51PM +0530, Ganapatrao Kulkarni escreveu: >> Signed-off-by: Ganapatrao Kulkarni > > Can you please consider to provide an example of such counters being > used, i.

Re: [PATCH] perf vendor events arm64: Update ThunderX2 implementation defined pmu core events

2018-07-31 Thread Ganapatrao Kulkarni
Hi Arnaldo, On Tue, Jul 31, 2018 at 10:59 PM, Arnaldo Carvalho de Melo wrote: > Em Tue, Jul 31, 2018 at 08:40:51PM +0530, Ganapatrao Kulkarni escreveu: >> Hi Arnaldo, >> >> On Tue, Jul 31, 2018 at 7:58 PM, Arnaldo Carvalho de Melo >> wrote: >> > Em T

Re: [PATCH v3] arm64: perf: Ignore exclude_hv when kernel is running in HYP

2017-05-15 Thread Ganapatrao Kulkarni
Hi Will, if this patch looks good, can it go in 4.12-rc2 as bug fix? or can it be queued for 4.13? On Tue, May 2, 2017 at 9:59 PM, Ganapatrao Kulkarni wrote: > commit d98ecdaca296 ("arm64: perf: Count EL2 events if the kernel is > running in HYP") returns -EINVAL when

Re: [PATCH v2] irqchip/gic-v3-its: Add workaround for ThunderX2 erratum #174

2018-01-19 Thread Ganapatrao Kulkarni
On Fri, Jan 19, 2018 at 5:53 PM, Marc Zyngier wrote: > On 18/01/18 05:28, Ganapatrao Kulkarni wrote: >> This erratum is observed on the ThunderX2 GICv3 ITS. When a >> MOVI command is used to change affinity of a LPI to a collection/cpu >> on another node, the LPI is not

[PATCH v2] irqchip/gic-v3-its: Add workaround for ThunderX2 erratum #174

2018-01-17 Thread Ganapatrao Kulkarni
after MOVI, there is a chance that we lose LPIs which are raised when the affinity is changed. So for now, adding workaround fix to disable inter node affinity change. Signed-off-by: Ganapatrao Kulkarni --- v2: Added workaround to avoid inter node affinity change. v1: Initial patch Documentation

[PATCH] perf vendor events arm64: Enable JSON events for ThunderX2 B0

2018-03-07 Thread Ganapatrao Kulkarni
There is MIDR change on ThunderX2 B0, adding an entry to mapfile to enable JSON events for B0. Signed-off-by: Ganapatrao Kulkarni --- tools/perf/pmu-events/arch/arm64/mapfile.csv | 1 + 1 file changed, 1 insertion(+) diff --git a/tools/perf/pmu-events/arch/arm64/mapfile.csv b/tools/perf/pmu

Re: [PATCH] perf vendor events arm64: Enable JSON events for ThunderX2 B0

2018-03-07 Thread Ganapatrao Kulkarni
Hi Will Cohen, On Wed, Mar 7, 2018 at 8:08 PM, Arnaldo Carvalho de Melo wrote: > Em Wed, Mar 07, 2018 at 09:32:05AM -0500, William Cohen escreveu: >> On 03/07/2018 06:08 AM, Ganapatrao Kulkarni wrote: >> > There is MIDR change on ThunderX2 B0, adding an entry to mapfile &

Re: [PATCH] perf vendor events arm64: Enable JSON events for ThunderX2 B0

2018-03-07 Thread Ganapatrao Kulkarni
On Thu, Mar 8, 2018 at 12:01 AM, William Cohen wrote: > On 03/07/2018 12:35 PM, Ganapatrao Kulkarni wrote: >> Hi Will Cohen, >> >> On Wed, Mar 7, 2018 at 8:08 PM, Arnaldo Carvalho de Melo >> wrote: >>> Em Wed, Mar 07, 2018 at 09:32:05AM -0500, William Cohen e

Re: [PATCH] perf vendor events arm64: Enable JSON events for ThunderX2 B0

2018-03-09 Thread Ganapatrao Kulkarni
Hi Arnaldo, can you please pull-in this patch? On Thu, Mar 8, 2018 at 9:44 AM, Ganapatrao Kulkarni wrote: > On Thu, Mar 8, 2018 at 12:01 AM, William Cohen wrote: >> On 03/07/2018 12:35 PM, Ganapatrao Kulkarni wrote: >>> Hi Will Cohen, >>> >>> On Wed, Mar 7,

Re: [PATCH v3 09/11] perf vendor events arm64: fixup ThunderX2 to use recommended events

2018-03-09 Thread Ganapatrao Kulkarni
Hi John, On Thu, Mar 8, 2018 at 4:28 PM, John Garry wrote: > This patch fixes the Cavium ThunderX2 JSON to use event definitions > from the ARMv8 recommended events. > > Cc: Ganapatrao Kulkarni > Signed-off-by: John Garry > --- > .../arch/arm64/cavium/thunderx2/co

Re: [PATCH] perf vendor events arm64: Enable JSON events for ThunderX2 B0

2018-03-09 Thread Ganapatrao Kulkarni
Hi Arnaldo, On Fri, Mar 9, 2018 at 11:03 PM, Arnaldo Carvalho de Melo wrote: > Em Fri, Mar 09, 2018 at 03:58:09PM +, Will Deacon escreveu: >> On Fri, Mar 09, 2018 at 11:34:15AM -0300, Arnaldo Carvalho de Melo wrote: >> > Em Fri, Mar 09, 2018 at 07:57:04PM +0530, Ganapatrao

Re: [PATCH] perf vendor events arm64: Enable JSON events for ThunderX2 B0

2018-03-09 Thread Ganapatrao Kulkarni
On Fri, Mar 9, 2018 at 11:32 PM, Arnaldo Carvalho de Melo wrote: > Em Fri, Mar 09, 2018 at 03:00:40PM -0300, Arnaldo Carvalho de Melo escreveu: >> Em Fri, Mar 09, 2018 at 11:15:16PM +0530, Ganapatrao Kulkarni escreveu: >> > On Fri, Mar 9, 2018 at 11:03 PM, Arnaldo Carvalho de

Re: [PATCH v11 0/2] Add ThunderX2 SoC Performance Monitoring Unit driver

2018-12-06 Thread Ganapatrao Kulkarni
On Thu, Dec 6, 2018 at 6:04 PM Will Deacon wrote: > > Hi Ganapat, > > On Thu, Dec 06, 2018 at 11:51:24AM +, Kulkarni, Ganapatrao wrote: > > This patchset adds PMU driver for Cavium's ThunderX2 SoC UNCORE devices. > > The SoC has PMU support in L3 cache controller (L3C) and in the > > DDR4 Memo

Re: [PATCH v5 2/2] ThunderX2: Add Cavium ThunderX2 SoC UNCORE PMU driver

2018-05-18 Thread Ganapatrao Kulkarni
On Thu, May 17, 2018 at 4:42 PM, John Garry wrote: > On 16/05/2018 05:55, Ganapatrao Kulkarni wrote: >> >> This patch adds a perf driver for the PMU UNCORE devices DDR4 Memory >> Controller(DMC) and Level 3 Cache(L3C). >> > > Hi, > > Just some codi

Re: [PATCH v6 2/2] ThunderX2: Add Cavium ThunderX2 SoC UNCORE PMU driver

2018-10-10 Thread Ganapatrao Kulkarni
Hi Suzuki, On Wed, Oct 10, 2018 at 3:22 PM Suzuki K Poulose wrote: > > Hi Ganapatrao, > > On 21/06/18 07:33, Ganapatrao Kulkarni wrote: > > This patch adds a perf driver for the PMU UNCORE devices DDR4 Memory > > Controller(DMC) and Level 3 Cache(L3C). > > >

Re: [PATCH] arm_pmu: Delete incorrect cache event mapping for some armv8_pmuv3 events.

2018-10-03 Thread Ganapatrao Kulkarni
Hi Will, can you please pull this patch? On Mon, Oct 1, 2018 at 10:09 PM Ganapatrao Kulkarni wrote: > > Hi Will, > > On Mon, Oct 1, 2018 at 7:58 PM Will Deacon wrote: > > > > Hi Ganapat, > > > > On Mon, Oct 01, 2018 at 10:07:43AM +, Kulkarni, Ganapatra

Re: [PATCH] arm_pmu: Delete incorrect cache event mapping for some armv8_pmuv3 events.

2018-10-04 Thread Ganapatrao Kulkarni
Hi Will, On Thu, Oct 4, 2018 at 5:51 PM Will Deacon wrote: > > Hi Ganapat, > > On Thu, Oct 04, 2018 at 11:12:09AM +0530, Ganapatrao Kulkarni wrote: > > can you please pull this patch? > > I still don't like the idea of just removing events like this, especial

Re: [PATCH v6 2/2] ThunderX2: Add Cavium ThunderX2 SoC UNCORE PMU driver

2018-10-08 Thread Ganapatrao Kulkarni
Hi Pranith, On Sat, Jul 7, 2018 at 11:22 AM Pranith Kumar wrote: > > Hi Ganapatrao, > > > On Wed, Jun 20, 2018 at 11:33 PM, Ganapatrao Kulkarni > wrote: > > > + > > +enum thunderx2_uncore_l3_events { > > + L3_EVENT_NONE, > > + L3_EVE

Re: [PATCH v4 1/2] perf: uncore: Adding documentation for ThunderX2 pmu uncore driver

2018-04-26 Thread Ganapatrao Kulkarni
On Fri, Apr 27, 2018 at 2:25 AM, Randy Dunlap wrote: > Hi, > > Just a few typo corrections... > > On 04/25/2018 02:00 AM, Ganapatrao Kulkarni wrote: >> Documentation for the UNCORE PMUs on Cavium's ThunderX2 SoC. >> The SoC has PMU support in its L3 cache con

Re: [PATCH v4 2/2] ThunderX2: Add Cavium ThunderX2 SoC UNCORE PMU driver

2018-05-15 Thread Ganapatrao Kulkarni
Hi Mark, On Sat, May 5, 2018 at 12:16 AM, Ganapatrao Kulkarni wrote: > Hi Mark, > > On Thu, Apr 26, 2018 at 4:29 PM, Mark Rutland wrote: >> Hi, >> >> On Wed, Apr 25, 2018 at 02:30:47PM +0530, Ganapatrao Kulkarni wrote: >>> + >>> +/* L3c and DMC

[PATCH v5 0/2] Add ThunderX2 SoC Performance Monitoring Unit driver

2018-05-15 Thread Ganapatrao Kulkarni
patch Ganapatrao Kulkarni (2): perf: uncore: Adding documentation for ThunderX2 pmu uncore driver ThunderX2: Add Cavium ThunderX2 SoC UNCORE PMU driver Documentation/perf/thunderx2-pmu.txt | 66 +++ drivers/perf/Kconfig | 8 + drivers/perf/Makefile| 1 + dr

[PATCH v5 1/2] perf: uncore: Adding documentation for ThunderX2 pmu uncore driver

2018-05-15 Thread Ganapatrao Kulkarni
Documentation for the UNCORE PMUs on Cavium's ThunderX2 SoC. The SoC has PMU support in its L3 cache controller (L3C) and in the DDR4 Memory Controller (DMC). Signed-off-by: Ganapatrao Kulkarni --- Documentation/perf/thunderx2-pmu.txt | 66 1 file ch

[PATCH v5 2/2] ThunderX2: Add Cavium ThunderX2 SoC UNCORE PMU driver

2018-05-15 Thread Ganapatrao Kulkarni
tiles of L3 cache. Each PMU supports up to 4 counters. All counters lack overflow interrupt and are sampled periodically. Signed-off-by: Ganapatrao Kulkarni --- drivers/perf/Kconfig | 8 + drivers/perf/Makefile| 1 + drivers/perf/thunderx2_pmu.c | 965

[PATCH v4 0/2] Add ThunderX2 SoC Performance Monitoring Unit driver

2018-04-25 Thread Ganapatrao Kulkarni
l v3: - fixed warning reported by kbuild robot v2: - rebased to 4.12-rc1 - Removed Arch VULCAN dependency. - update SMC call parameters as per latest firmware. v1: -Initial patch Ganapatrao Kulkarni (2): perf: uncore: Adding documentation for ThunderX2 pmu uncore d

[PATCH v4 2/2] ThunderX2: Add Cavium ThunderX2 SoC UNCORE PMU driver

2018-04-25 Thread Ganapatrao Kulkarni
tiles of L3 cache. Each PMU supports up to 4 counters. All counters lack overflow interrupt and are sampled periodically. Signed-off-by: Ganapatrao Kulkarni --- drivers/perf/Kconfig | 8 + drivers/perf/Makefile| 1 + drivers/perf/thunderx2_pmu.c | 958

[PATCH v4 1/2] perf: uncore: Adding documentation for ThunderX2 pmu uncore driver

2018-04-25 Thread Ganapatrao Kulkarni
Documentation for the UNCORE PMUs on Cavium's ThunderX2 SoC. The SoC has PMU support in its L3 cache controller (L3C) and in the DDR4 Memory Controller (DMC). Signed-off-by: Ganapatrao Kulkarni --- Documentation/perf/thunderx2-pmu.txt | 66 1 file ch

Re: [PATCH] iommu/iova: Update cached node pointer when current node fails to get any free IOVA

2018-04-26 Thread Ganapatrao Kulkarni
Hi Robin, On Mon, Apr 23, 2018 at 11:11 PM, Ganapatrao Kulkarni wrote: > On Mon, Apr 23, 2018 at 10:07 PM, Robin Murphy wrote: >> On 19/04/18 18:12, Ganapatrao Kulkarni wrote: >>> >>> The performance drop is observed with long hours iperf testing using 40G >>&g

[PATCH v2] irqchip, gicv3-its, numa: Workaround for Cavium ThunderX erratum 23144

2015-08-25 Thread Ganapatrao Kulkarni
outed to a redistributors present on a foreign node. v2: updatated as per Marc Zyngier's review comments. Signed-off-by: Ganapatrao Kulkarni Signed-off-by: Robert Richter --- drivers/irqchip/irq-gic-v3-its.c | 53 +--- 1 file changed, 44 insertions(

[PATCH v2 RESEND] irqchip, gicv3-its, numa: Workaround for Cavium ThunderX erratum 23144

2015-08-25 Thread Ganapatrao Kulkarni
This implements a workaround for gicv3-its erratum 23144 on Cavium's ThunderX dual-socket platforms, where LPI cannot be routed to a redistributors present on a foreign node. Signed-off-by: Ganapatrao Kulkarni Signed-off-by: Robert Richter --- The patch below is on top of Robert's re

Re: [PATCH 3/3] arm64/numa: fix type info

2016-05-26 Thread Ganapatrao Kulkarni
On Wed, May 25, 2016 at 7:43 PM, Zhen Lei wrote: > numa_init(of_numa_init) may returned error because of numa configuration > error. So "No NUMA configuration found" is inaccurate. In fact, specific > configuration error information can be immediately printed by the > testing branch. So "No NUMA..

Re: [PATCH v12 4/5] arm64, numa: Add NUMA support for arm64 platforms.

2016-02-26 Thread Ganapatrao Kulkarni
On Sat, Feb 27, 2016 at 1:21 AM, David Daney wrote: > On 02/26/2016 10:53 AM, Will Deacon wrote: > [...] >>> >>> diff --git a/arch/arm64/mm/numa.c b/arch/arm64/mm/numa.c >>> new file mode 100644 >>> index 000..604e886 >>> --- /dev/null >>> +++ b/arch/arm64/mm/numa.c >>> @@ -0,0 +1,403 @@ > > [

[PATCH v3] irqchip, gicv3-its, numa: Workaround for Cavium thunderx erratum 23144

2016-02-16 Thread Ganapatrao Kulkarni
ernel/msg443462.html v2: updatated as per Marc Zyngier's review comments. Signed-off-by: Ganapatrao Kulkarni --- arch/arm64/Kconfig | 9 drivers/irqchip/irq-gic-v3-its.c | 46 ++-- 2 files changed, 53 insertions(+), 2 deletion

Re: [PATCH v5 12/14] arm64, acpi, numa: NUMA support based on SRAT and SLIT

2016-04-20 Thread Ganapatrao Kulkarni
to Proximity Domain mappings and memory >> ranges to Proximity Domain mapping. SLIT has the information of inter >> node distances(relative number for access latency). >> >> Signed-off-by: Hanjun Guo >> Signed-off-by: Ganapatrao Kulkarni >> [rrich...@cavium.com Re

[PATCH] arm64: Fix the ptep_set_wrprotect() to set PTE_DIRTY if (PTE_DBM && !PTE_RDONLY)

2016-03-08 Thread Ganapatrao Kulkarni
ote: this patch is not tested on platform which supports AFDBM. Signed-off-by: Ganapatrao Kulkarni --- arch/arm64/include/asm/pgtable.h | 24 ++-- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtab

Re: [PATCH] arm64: Fix the ptep_set_wrprotect() to set PTE_DIRTY if (PTE_DBM && !PTE_RDONLY)

2016-03-09 Thread Ganapatrao Kulkarni
On Wed, Mar 9, 2016 at 3:36 PM, Catalin Marinas wrote: > On Wed, Mar 09, 2016 at 10:32:48AM +0530, Ganapatrao Kulkarni wrote: >> Commit 2f4b829c625e ("arm64: Add support for hardware updates of the >> access and dirty pte bits") introduced support for handling hardware

Re: [PATCH] arm64: Fix the ptep_set_wrprotect() to set PTE_DIRTY if (PTE_DBM && !PTE_RDONLY)

2016-03-09 Thread Ganapatrao Kulkarni
On Wed, Mar 9, 2016 at 9:33 PM, Catalin Marinas wrote: > On Wed, Mar 09, 2016 at 05:17:39PM +0530, Ganapatrao Kulkarni wrote: >> On Wed, Mar 9, 2016 at 3:36 PM, Catalin Marinas >> wrote: >> > On Wed, Mar 09, 2016 at 10:32:48AM +0530, Ganapatrao Kulkarni wrote: >>

Re: [PATCH] arm64: Fix the ptep_set_wrprotect() to set PTE_DIRTY if (PTE_DBM && !PTE_RDONLY)

2016-03-09 Thread Ganapatrao Kulkarni
On Wed, Mar 9, 2016 at 11:13 PM, Ganapatrao Kulkarni wrote: > On Wed, Mar 9, 2016 at 9:33 PM, Catalin Marinas > wrote: >> On Wed, Mar 09, 2016 at 05:17:39PM +0530, Ganapatrao Kulkarni wrote: >>> On Wed, Mar 9, 2016 at 3:36 PM, Catalin Marinas >>> wrote: >&

Re: [PATCH] perf cgroups: Don't rotate events for cgroups unnecessarily

2019-08-23 Thread Ganapatrao Kulkarni
Hi, We are seeing regression with our uncore perf driver(Marvell's ThunderX2, ARM64 server platform) on 5.3-Rc1. After bisecting, it turned out to be this patch causing the issue. Test case: Load module and run perf for more than 4 events( we have 4 counters, event multiplexing takes place for mo

Re: [PATCH] perf cgroups: Don't rotate events for cgroups unnecessarily

2019-08-23 Thread Ganapatrao Kulkarni
> > On Fri, Aug 23, 2019 at 04:13:46PM +0530, Ganapatrao Kulkarni wrote: > > > We are seeing regression with our uncore perf driver(Marvell's > > ThunderX2, ARM64 server platform) on 5.3-Rc1. > > After bisecting, it turned out to be this patch causing the issue. &

Re: [PATCH] perf cgroups: Don't rotate events for cgroups unnecessarily

2019-09-18 Thread Ganapatrao Kulkarni
On Fri, Aug 23, 2019 at 6:33 PM Peter Zijlstra wrote: > > On Fri, Aug 23, 2019 at 06:26:34PM +0530, Ganapatrao Kulkarni wrote: > > On Fri, Aug 23, 2019 at 5:29 PM Peter Zijlstra wrote: > > > On Fri, Aug 23, 2019 at 04:13:46PM +0530, Ganapatrao Kulkarni wrote: > &

Re: [PATCH] perf cgroups: Don't rotate events for cgroups unnecessarily

2019-10-14 Thread Ganapatrao Kulkarni
Hi Peter, On Wed, Sep 18, 2019 at 12:51 PM Ganapatrao Kulkarni wrote: > > On Fri, Aug 23, 2019 at 6:33 PM Peter Zijlstra wrote: > > > > On Fri, Aug 23, 2019 at 06:26:34PM +0530, Ganapatrao Kulkarni wrote: > > > On Fri, Aug 23, 2019 at 5:29 PM Peter Zijlstra > >

Re: [PATCH v6 2/2] drivers/perf: Add CCPI2 PMU support in ThunderX2 UNCORE driver.

2019-10-18 Thread Ganapatrao Kulkarni
On Fri, Oct 18, 2019 at 2:08 PM John Garry wrote: > > On 18/10/2019 05:21, Ganapatrao Kulkarni wrote: > > Hi Will, > > > > On Thu, Oct 17, 2019 at 9:17 PM Will Deacon wrote: > >> > >> On Thu, Oct 17, 2019 at 12:38:51PM +0530, Ganapatrao Kulkarni wrote:

[PATCH v3 0/2] Add CCPI2 PMU support

2019-07-23 Thread Ganapatrao Kulkarni
Add Cavium Coherent Processor Interconnect (CCPI2) PMU support in ThunderX2 Uncore driver. v3: Rebased to 5.3-rc1 v2: Updated with review comments [1] [1] https://lkml.org/lkml/2019/6/14/965 v1: initial patch Ganapatrao Kulkarni (2): Documentation: perf: Update documentation for ThunderX2

Re: [PATCH 2/2] drivers/perf: Add CCPI2 PMU support in ThunderX2 UNCORE driver.

2019-06-27 Thread Ganapatrao Kulkarni
Hi will, On Thu, Jun 27, 2019 at 3:27 PM Will Deacon wrote: > > Hi Ganapat, > > On Fri, Jun 14, 2019 at 05:42:46PM +, Ganapatrao Kulkarni wrote: > > CCPI2 is a low-latency high-bandwidth serial interface for connecting > > ThunderX2 processors. This patch adds suppo

Re: [PATCH v3] irqchip: gicv3-its: Use NUMA aware memory allocation for ITS tables

2019-01-10 Thread Ganapatrao Kulkarni
Hi Shameer, Patch looks OK to me, please feel free to add, Reviewed-by: Ganapatrao Kulkarni On Thu, Dec 13, 2018 at 5:25 PM Marc Zyngier wrote: > > On 13/12/2018 10:59, Shameer Kolothum wrote: > > From: Shanker Donthineni > > > > The NUMA node information is visib

Re: [PATCH] arm_pmu: Delete incorrect cache event mapping for some armv8_pmuv3 events.

2018-10-01 Thread Ganapatrao Kulkarni
1D_CACHE_REFILL. This is incorrect, > > since L1D_CACHE_REFILL counts both load and store misses. > > Similarly the events L1-dcache-loads, L1-dcache-stores, dTLB-load-misses > > and dTLB-loads are wrongly mapped. Hence Deleting all these cache events > > from armv8_pmuv3

Re: [PATCH] iommu/iova: Update cached node pointer when current node fails to get any free IOVA

2018-05-20 Thread Ganapatrao Kulkarni
On Thu, Apr 26, 2018 at 3:15 PM, Ganapatrao Kulkarni wrote: > Hi Robin, > > On Mon, Apr 23, 2018 at 11:11 PM, Ganapatrao Kulkarni > wrote: >> On Mon, Apr 23, 2018 at 10:07 PM, Robin Murphy wrote: >>> On 19/04/18 18:12, Ganapatrao Kulkarni wrote: >>>> >

Re: [PATCH v4 2/2] ThunderX2: Add Cavium ThunderX2 SoC UNCORE PMU driver

2018-05-21 Thread Ganapatrao Kulkarni
Hi Mark, On Mon, May 21, 2018 at 4:25 PM, Mark Rutland wrote: > On Sat, May 05, 2018 at 12:16:13AM +0530, Ganapatrao Kulkarni wrote: >> On Thu, Apr 26, 2018 at 4:29 PM, Mark Rutland wrote: >> > On Wed, Apr 25, 2018 at 02:30:47PM +0530, Ganapatrao Kulkarni wrote: > >>

Re: [PATCH v4 2/2] ThunderX2: Add Cavium ThunderX2 SoC UNCORE PMU driver

2018-05-21 Thread Ganapatrao Kulkarni
On Mon, May 21, 2018 at 4:10 PM, Mark Rutland wrote: > On Mon, May 21, 2018 at 11:37:12AM +0100, Mark Rutland wrote: >> Hi Ganapat, >> >> >> Sorry for the delay in replying; I was away most of last week. >> >> On Tue, May 15, 2018 at 04:03:19PM +0530, Ganapa

Re: [PATCH v2 00/11] perf events patches for improved ARM64 support

2018-03-02 Thread Ganapatrao Kulkarni
Hi John, On Fri, Mar 2, 2018 at 1:54 PM, John Garry wrote: > On 27/02/2018 09:50, Jiri Olsa wrote: >> >> On Sat, Feb 24, 2018 at 12:05:21AM +0800, John Garry wrote: >>> >>> This patchset adds support for some perf events features, >>> targeted at ARM64, implemented in a generic fashion. >>> >>> T

Re: [PATCH v2 00/11] perf events patches for improved ARM64 support

2018-03-02 Thread Ganapatrao Kulkarni
Hi John, On Fri, Mar 2, 2018 at 9:35 PM, William Cohen wrote: > On 03/02/2018 03:24 AM, John Garry wrote: >> On 27/02/2018 09:50, Jiri Olsa wrote: >>> On Sat, Feb 24, 2018 at 12:05:21AM +0800, John Garry wrote: This patchset adds support for some perf events features, targeted at ARM64,

Re: [PATCH] iommu/iova: Update cached node pointer when current node fails to get any free IOVA

2018-04-23 Thread Ganapatrao Kulkarni
On Mon, Apr 23, 2018 at 10:07 PM, Robin Murphy wrote: > On 19/04/18 18:12, Ganapatrao Kulkarni wrote: >> >> The performance drop is observed with long hours iperf testing using 40G >> cards. This is mainly due to long iterations in finding the free iova >> range in 32

Re: [PATCH v4 2/2] ThunderX2: Add Cavium ThunderX2 SoC UNCORE PMU driver

2018-05-04 Thread Ganapatrao Kulkarni
Hi Mark, On Thu, Apr 26, 2018 at 4:29 PM, Mark Rutland wrote: > Hi, > > On Wed, Apr 25, 2018 at 02:30:47PM +0530, Ganapatrao Kulkarni wrote: >> + >> +/* L3c and DMC has 16 and 8 channels per socket respectively. >> + * Each Channel supports UNCORE PMU device and consi

[PATCH] iommu/iova: Update cached node pointer when current node fails to get any free IOVA

2018-04-19 Thread Ganapatrao Kulkarni
update cached32_node to itself. From now on, walking over 32-bit range is more expensive. This patch adds fix to update cached node to leaf node when there are no iova free range left, which avoids unnecessary long iterations. Signed-off-by: Ganapatrao Kulkarni --- drivers/iommu/iova.c | 6 ++ 1

Re: fd3e45436660 ("ACPI / NUMA: ia64: Parse all entries of SRAT memory affinity table")

2018-05-09 Thread Ganapatrao Kulkarni
Hi Michal On Wed, May 9, 2018 at 5:54 PM, Michal Hocko wrote: > On Wed 11-04-18 12:48:32, Michal Hocko wrote: >> Hi, >> my attention was brought to the %subj commit and either I am missing >> something or the patch is quite dubious. What is it actually trying to >> fix? If a BIOS/FW provides mor

Re: fd3e45436660 ("ACPI / NUMA: ia64: Parse all entries of SRAT memory affinity table")

2018-05-09 Thread Ganapatrao Kulkarni
On Wed, May 9, 2018 at 6:26 PM, Michal Hocko wrote: > On Wed 09-05-18 18:07:16, Ganapatrao Kulkarni wrote: >> Hi Michal >> >> >> On Wed, May 9, 2018 at 5:54 PM, Michal Hocko wrote: >> > On Wed 11-04-18 12:48:32, Michal Hocko wrote: >> >> Hi, >&g

Re: fd3e45436660 ("ACPI / NUMA: ia64: Parse all entries of SRAT memory affinity table")

2018-05-10 Thread Ganapatrao Kulkarni
On Thu, May 10, 2018 at 1:00 PM, Michal Hocko wrote: > On Thu 10-05-18 08:27:35, Ganapatrao Kulkarni wrote: >> On Wed, May 9, 2018 at 6:26 PM, Michal Hocko wrote: >> > On Wed 09-05-18 18:07:16, Ganapatrao Kulkarni wrote: >> >> Hi Michal >> >> >>

Re: getting oom/stalls for ltp test cpuset01 with latest/4.9 kernel

2017-01-12 Thread Ganapatrao Kulkarni
On Thu, Jan 12, 2017 at 4:40 PM, Vlastimil Babka wrote: > On 01/11/2017 05:46 PM, Michal Hocko wrote: >> >> On Wed 11-01-17 21:52:29, Ganapatrao Kulkarni wrote: >> >>> [ 2398.169391] Node 1 Normal: 951*4kB (UME) 1308*8kB (UME) 1034*16kB >>> (UME) 742*32kB (

Re: getting oom/stalls for ltp test cpuset01 with latest/4.9 kernel

2017-01-16 Thread Ganapatrao Kulkarni
On Fri, Jan 13, 2017 at 2:36 PM, Vlastimil Babka wrote: > On 01/13/2017 05:35 AM, Ganapatrao Kulkarni wrote: >> On Thu, Jan 12, 2017 at 4:40 PM, Vlastimil Babka wrote: >>> On 01/11/2017 05:46 PM, Michal Hocko wrote: >>>> >>>> On Wed 11

[PATCH] arm64: Enable workaround for Cavium erratum 27456 on thunderx-81xx

2016-07-06 Thread Ganapatrao Kulkarni
Cavium erratum 27456 commit 104a0c02e8b1 ("arm64: Add workaround for Cavium erratum 27456") is applicable for thunderx-81xx pass1.0 SoC as well. Adding code to enable to 81xx. Signed-off-by: Ganapatrao Kulkarni Reviewed-by: Andrew Pinski --- arch/arm64/include/asm/cputype.h | 2 ++

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

2016-11-30 Thread Ganapatrao Kulkarni
Hi Eric, in you repo "https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3"; there is 11th patch "pci: Enable overrides for missing ACS capabilities" is this patch part of some other series? thanks Ganapat On Wed, Nov 30, 2016 at 3:19 PM, Auger Eric wrote: > Hi, > > On 15/11/2016 14:09

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

2016-11-30 Thread Ganapatrao Kulkarni
On Wed, Nov 30, 2016 at 3:44 PM, Auger Eric wrote: > Hi Ganapat, > > On 30/11/2016 11:04, Ganapatrao Kulkarni wrote: >> Hi Eric, >> >> in you repo "https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3"; >> there is 11th patch "pci: Enab

getting oom/stalls for ltp test cpuset01 with latest/4.9 kernel

2017-01-11 Thread Ganapatrao Kulkarni
Hi, we are seeing OOM/stalls messages when we run ltp cpuset01(cpuset01 -I 360) test for few minutes, even through the numa system has adequate memory on both nodes. this we have observed same on both arm64/thunderx numa and on x86 numa system! using latest ltp from master branch version 2016092

Re: [PATCH v4 12/14] arm64/numa: remove some useless code

2016-06-07 Thread Ganapatrao Kulkarni
On Tue, Jun 7, 2016 at 1:38 PM, Zhen Lei wrote: > 1. Currently only cpu0 set on cpu_possible_mask and percpu areas have not >been initialized. > 2. No reason to limit cpu0 must belongs to node0. even smp init assumes cpu0/boot processor. is this patch tested on any hardware? can you describe

Re: [PATCH v4 11/14] arm64/numa: support HAVE_MEMORYLESS_NODES

2016-06-07 Thread Ganapatrao Kulkarni
On Tue, Jun 7, 2016 at 1:38 PM, Zhen Lei wrote: > Some numa nodes may have no memory. For example: > 1. cpu0 on node0 > 2. cpu1 on node1 > 3. device0 access the momory from node0 and node1 take the same time. i am wondering, if access to both nodes is same, then why you need numa. the example you

Re: [PATCH v4 11/14] arm64/numa: support HAVE_MEMORYLESS_NODES

2016-06-07 Thread Ganapatrao Kulkarni
On Tue, Jun 7, 2016 at 6:27 PM, Leizhen (ThunderTown) wrote: > > > On 2016/6/7 16:31, Ganapatrao Kulkarni wrote: >> On Tue, Jun 7, 2016 at 1:38 PM, Zhen Lei wrote: >>> Some numa nodes may have no memory. For example: >>> 1. cpu0 on node0 >>> 2. cpu1 on n

Re: [PATCH v4 11/14] arm64/numa: support HAVE_MEMORYLESS_NODES

2016-06-07 Thread Ganapatrao Kulkarni
On Wed, Jun 8, 2016 at 7:46 AM, Leizhen (ThunderTown) wrote: > > > On 2016/6/7 22:01, Ganapatrao Kulkarni wrote: >> On Tue, Jun 7, 2016 at 6:27 PM, Leizhen (ThunderTown) >> wrote: >>> >>> >>> On 2016/6/7 16:31, Ganapatrao Kulkarni wrote: >>&g

  1   2   3   >