Daan, thanks for info. And one final question - is it possible to change Public vlan/range from untagged to tagged - editing "vlan" table and making change, does not really make changes after I restart VPC router... THanks
On 29 May 2014 10:15, Daan Hoogland <daan.hoogl...@gmail.com> wrote: > Andrija, we plan to release it soon. It depends on how quickly issues > in it are fixed. The branch has been created, so you could test it and > report on it if you have an env to spare. > > On Thu, May 29, 2014 at 10:09 AM, Andrija Panic <andrija.pa...@gmail.com> > wrote: > > Will try, thx. Not sure good question - when is 4.4 scheduled to be > > releases - few months, or more ? > > > > Thanks > > > > > > On 29 May 2014 06:15, Jayapal Reddy Uradi <jayapalreddy.ur...@citrix.com > >wrote: > > > >> Hi Andrija, > >> > >> Same issue with public vlan tagged got fixed, CLOUDSTACK-5505. > >> > >> Thanks, > >> Jayapal > >> > >> On 29-May-2014, at 9:38 AM, Jayapal Reddy Uradi < > >> jayapalreddy.ur...@citrix.com> > >> wrote: > >> > >> > Hi Adrija, > >> > > >> > From the logs, the public subnet is untagged. > >> > I think this issue is coming for the untagged public vlan in 4.3. > >> > > >> > > >> > 1. > >> > > >> > {"com.cloud.agent.api.PlugNicCommand":{"nic":{"deviceId":1,"networkRateMbps":99999,"defaultNic":true,"uuid":"e6b734d4-3302-4113-8ec7-5c205c90959a","ip":"46.232.180.248","netmask":"255.255.255.0","gateway":"46.232.180.1","mac":"06:5e:e8:00:00:27","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://untagged","isolationUri":"vlan://untagged","isSecurityGroupEnabled":false,"name":"breth1-500"}, > >> > 2. > >> > > >> > 3. > >> > > >> > "instanceName":"r-779-VM","vmType":"DomainRouter","wait":0}},{"com.cloud.agent.api.routing.IpAssocVpcCommand":{"ipAddresses":[{"accountId":2,"publicIp":"46.232.180.248","sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":false,"broadcastUri":"untagged","vlanGateway":"46.232.180.1","vlanNetmask":"255.255.255.0","vifMacAddress":"06:5e:e8:00:00:27","networkRate":99999,"trafficType":"Public","networkName":"breth1-500"}],"accessDetails": > >> > > >> > > >> > From the logs VR logs, the ipassoc script got the interface id as > null. > >> > May 28 12:37:33 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> ethnull to appear, 0 seconds > >> > > >> > Thanks, > >> > Jayapal > >> > > >> > On 29-May-2014, at 1:08 AM, Andrija Panic <andrija.pa...@gmail.com > >> <mailto:andrija.pa...@gmail.com>> wrote: > >> > > >> > Thanks Daan, > >> > > >> > my problem is that I'm in production for 3rd day now, and restoring DB > >> and > >> > downgrading back to 4.2.1 doesn't seem as option for me at the moment, > >> > since I would loose new acounts and single VMs, etc... > >> > > >> > Thanks, > >> > Andrija > >> > > >> > > >> > On 28 May 2014 21:34, Daan Hoogland <daan.hoogl...@gmail.com<mailto: > >> daan.hoogl...@gmail.com>> wrote: > >> > > >> > Andrija, > >> > > >> > nevertheless it sounds familiar. I will be back in the office on > >> > monday and ask around. > >> > > >> > On Wed, May 28, 2014 at 9:23 PM, Andrija Panic < > andrija.pa...@gmail.com > >> <mailto:andrija.pa...@gmail.com>> > >> > wrote: > >> > Hi Daan, > >> > > >> > I don't think this is my issue, at least I don't make use of private > >> > gateway - this is just simple as: create new VPC from scratch - > Public > >> > IP > >> > is not assigned to VR eth1 interface inside VR... > >> > > >> > I have filed the bug: > >> > https://issues.apache.org/jira/browse/CLOUDSTACK-6801 > >> > > >> > This same thing happened previously to Andrei Mikhailovsky: > >> > > >> > > >> > http://mail-archives.apache.org/mod_mbox/cloudstack-users/201405.mbox/%3C33347835.250.1399336340785.JavaMail.andrei@tuchka%3Eand > >> > it is not resolved > >> > > >> > Thanks, > >> > > >> > Andrija > >> > > >> > > >> > On 28 May 2014 21:01, Daan Hoogland <daan.hoogl...@gmail.com> wrote: > >> > > >> > Andrija, > >> > > >> > this sound like something we seen as well. > >> > can you check if this is it : > >> > https://issues.apache.org/jira/browse/CLOUDSTACK-6485 > >> > > >> > thanks, > >> > Daan > >> > > >> > On Wed, May 28, 2014 at 3:30 PM, Andrija Panic < > andrija.pa...@gmail.com > >> > > >> > wrote: > >> > Hi there, > >> > > >> > I'm having big time problems with Public IP missing from VPC VR's > >> > eth1, > >> > after upgrade to ACS 4.3.1 - did not found this filed as bug so > >> > far...and > >> > it worked all fine on ACS 4.2.1. > >> > > >> > No help so far from user mailing list... > >> > > >> > Below is a detailed explanation, and logs from inside VR, and from > >> > management (all fine with management logs...) > >> > > >> > If anybody can help, I would very much appriciate this, since now I > >> > have > >> > bunch fo VPC unoperational... > >> > > >> > Thanks > >> > > >> > ---------- Forwarded message ---------- > >> > From: Andrija Panic <andrija.pa...@gmail.com> > >> > Date: 28 May 2014 14:50 > >> > Subject: Re: VPC's VR missing public NIC eth1 > >> > To: us...@cloudstack.apache.org > >> > > >> > > >> > and as I said eth1 is present: > >> > > >> > root@r-794-VM:~# cat /proc/net/dev > >> > Inter-| Receive | > >> > Transmit > >> > face |bytes packets errs drop fifo frame compressed > >> > multicast|bytes > >> > packets errs drop fifo colls carrier compressed > >> > eth3: 11484 131 0 0 0 0 0 0 > >> > 11590 > >> > 131 0 0 0 0 0 0 > >> > lo: 214 2 0 0 0 0 0 0 > >> > 214 > >> > 2 0 0 0 0 0 0 > >> > eth2: 32970 544 0 0 0 0 0 0 > >> > 2084 > >> > 24 0 0 0 0 0 0 > >> > eth1: 0 0 0 0 0 0 0 0 > >> > 0 > >> > 0 0 0 0 0 0 0 > >> > eth0: 150207 1319 0 0 0 0 0 0 > >> > 264232 > >> > 1180 0 0 0 0 0 0 > >> > > >> > > >> > On 28 May 2014 14:47, Andrija Panic <andrija.pa...@gmail.com> wrote: > >> > > >> > Also, from /var/log/messages/ inside VR: > >> > > >> > This is a major show stopper - all our VPCs are unusable complete. > >> > Anybody... ? > >> > > >> > May 28 12:37:33 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> > ethnull to appear, 0 seconds > >> > May 28 12:37:34 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> > ethnull to appear, 1 seconds > >> > May 28 12:37:35 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> > ethnull to appear, 2 seconds > >> > May 28 12:37:36 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> > ethnull to appear, 3 seconds > >> > May 28 12:37:37 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> > ethnull to appear, 4 seconds > >> > May 28 12:37:38 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> > ethnull to appear, 5 seconds > >> > May 28 12:37:39 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> > ethnull to appear, 6 seconds > >> > May 28 12:37:40 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> > ethnull to appear, 7 seconds > >> > May 28 12:37:41 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> > ethnull to appear, 8 seconds > >> > May 28 12:37:42 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> > ethnull to appear, 9 seconds > >> > May 28 12:37:43 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> > ethnull to appear, 10 seconds > >> > May 28 12:37:44 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> > ethnull to appear, 11 seconds > >> > May 28 12:37:45 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> > ethnull to appear, 12 seconds > >> > May 28 12:37:46 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> > ethnull to appear, 13 seconds > >> > May 28 12:37:47 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> > ethnull to appear, 14 seconds > >> > May 28 12:37:48 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> > ethnull to appear, 15 seconds > >> > May 28 12:37:49 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> > ethnull to appear, 16 seconds > >> > May 28 12:37:50 r-794-VM cloud: vpc_ipassoc.sh:interface ethnull > >> > never > >> > appeared > >> > May 28 12:37:50 r-794-VM cloud: vpc_ipassoc.sh:Adding ip > >> > 46.232.180.246 > >> > on > >> > interface ethnull > >> > May 28 12:37:50 r-794-VM cloud: vpc_ipassoc.sh:Add routing > >> > 46.232.180.246 > >> > on interface ethnull > >> > May 28 12:37:50 r-794-VM cloud: vpc_privateGateway.sh:Added SourceNAT > >> > 46.232.180.246 on interface ethnull > >> > May 28 12:37:50 r-794-VM cloud: vpc_snat.sh:Added SourceNAT > >> > 46.232.180.246 > >> > on interface eth1 > >> > > >> > > >> > > >> > > >> > On 28 May 2014 12:59, Andrija Panic <andrija.pa...@gmail.com> wrote: > >> > > >> > Defined eth1 manually inside /etc/network/interfaces inside VPC's > >> > VR. > >> > iface eth1 inet static > >> > address 46.232.180.246 > >> > netmask 255.255.255.0 > >> > > >> > ifup eth1 > >> > ip route add default via 46.232.180.1 > >> > > >> > so now VR works fine (have access to internet) > >> > > >> > But again, adding new IP to VR, and enabling static NAT is > >> > failing... > >> > That is, geting new IP works fine (just associated with account) > >> > But enabling static NAT fails, due to "resource unavailable" > >> > > >> > Here are management logs: > >> > 2014-05-28 12:57:00,716 WARN [c.c.n.r.RulesManagerImpl] > >> > (catalina-exec-22:ctx-537ac57b ctx-8c44c786) Failed to create static > >> > nat > >> > rule due to > >> > com.cloud.exception.ResourceUnavailableException: Resource > >> > [DataCenter:1] > >> > is unreachable: Unable to apply static nat rules on router > >> > at > >> > > >> > > >> > > >> > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3915) > >> > at > >> > > >> > > >> > > >> > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyStaticNats(VirtualNetworkApplianceManagerImpl.java:3963) > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > >> > Method) > >> > at > >> > > >> > > >> > > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > >> > at > >> > > >> > > >> > > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> > at java.lang.reflect.Method.invoke(Method.java:622) > >> > at > >> > > >> > > >> > > >> > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) > >> > at > >> > > >> > > >> > > >> > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) > >> > at > >> > > >> > > >> > > >> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) > >> > at > >> > > >> > > >> > > >> > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) > >> > at > >> > > >> > > >> > > >> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) > >> > at > >> > > >> > > >> > > >> > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) > >> > at com.sun.proxy.$Proxy240.applyStaticNats(Unknown Source) > >> > at > >> > > >> > > >> > > >> > com.cloud.network.element.VirtualRouterElement.applyStaticNats(VirtualRouterElement.java:650) > >> > at > >> > > >> > > >> > > >> > com.cloud.network.IpAddressManagerImpl.applyStaticNats(IpAddressManagerImpl.java:1762) > >> > at > >> > > >> > > >> > > >> > com.cloud.network.rules.RulesManagerImpl.applyStaticNatForIp(RulesManagerImpl.java:1324) > >> > at > >> > > >> > > >> > > >> > com.cloud.network.rules.RulesManagerImpl.enableStaticNat(RulesManagerImpl.java:602) > >> > at > >> > > >> > > >> > > >> > com.cloud.network.rules.RulesManagerImpl.enableStaticNat(RulesManagerImpl.java:446) > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > >> > Method) > >> > at > >> > > >> > > >> > > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > >> > at > >> > > >> > > >> > > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> > at java.lang.reflect.Method.invoke(Method.java:622) > >> > at > >> > > >> > > >> > > >> > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) > >> > at > >> > > >> > > >> > > >> > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) > >> > at > >> > > >> > > >> > > >> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) > >> > at > >> > > >> > > >> > > >> > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:50) > >> > at > >> > > >> > > >> > > >> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) > >> > at > >> > > >> > > >> > > >> > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) > >> > at > >> > > >> > > >> > > >> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) > >> > at > >> > > >> > > >> > > >> > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) > >> > at com.sun.proxy.$Proxy88.enableStaticNat(Unknown Source) > >> > at > >> > > >> > > >> > > >> > org.apache.cloudstack.api.command.user.nat.EnableStaticNatCmd.execute(EnableStaticNatCmd.java:129) > >> > at > >> > com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161) > >> > at com.cloud.api.ApiServer.queueCommand(ApiServer.java:531) > >> > at com.cloud.api.ApiServer.handleRequest(ApiServer.java:374) > >> > at > >> > > >> > com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:323) > >> > at com.cloud.api.ApiServlet.access$000(ApiServlet.java:53) > >> > at com.cloud.api.ApiServlet$1.run(ApiServlet.java:115) > >> > at > >> > > >> > > >> > > >> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > >> > at > >> > > >> > > >> > > >> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > >> > at > >> > > >> > > >> > > >> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > >> > at > >> > com.cloud.api.ApiServlet.processRequest(ApiServlet.java:112) > >> > at com.cloud.api.ApiServlet.doGet(ApiServlet.java:74) > >> > at > >> > javax.servlet.http.HttpServlet.service(HttpServlet.java:617) > >> > at > >> > javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > >> > at > >> > > >> > > >> > > >> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) > >> > at > >> > > >> > > >> > > >> > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > >> > at > >> > > >> > > >> > > >> > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) > >> > at > >> > > >> > > >> > > >> > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) > >> > at > >> > > >> > > >> > > >> > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) > >> > at > >> > > >> > > >> > > >> > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > >> > at > >> > > >> > > >> > > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555) > >> > at > >> > > >> > > >> > > >> > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > >> > at > >> > > >> > > >> > > >> > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) > >> > at > >> > > >> > > >> > > >> > org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889) > >> > at > >> > > >> > > >> > > >> > org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721) > >> > at > >> > > >> > > >> > > >> > org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274) > >> > at > >> > > >> > > >> > > >> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) > >> > at > >> > > >> > > >> > > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > >> > at java.lang.Thread.run(Thread.java:701) > >> > > >> > > >> > > >> > > >> > On 28 May 2014 00:58, Andrija Panic <andrija.pa...@gmail.com> > >> > wrote: > >> > > >> > Hi Jayapal, > >> > > >> > eth1 seems present (lspci and virsh comfirmed), but not started > >> > inside > >> > VPC's VR - (VR used for Shared Network is fine)... > >> > I could confirm by virsh that is is plugged inside appropriate > >> > bridge > >> > breth1-500 (check management logs...) > >> > > >> > management log while createing new VPC (VR) - > >> > http://pastebin.com/s77nu5Ei > >> > The public IP is there, so command is fine for creating it I > >> > guess... > >> > > >> > VR's /var/log/cloud.log after rebooting VR from CS GUI > >> > Tue May 27 22:46:58 UTC 2014 Executing cloud-early-config > >> > Tue May 27 22:46:58 UTC 2014 Detected that we are running inside > >> > kvm > >> > guest > >> > Tue May 27 22:46:59 UTC 2014 Patching cloud service > >> > Tue May 27 22:47:00 UTC 2014 Updating log4j-cloud.xml > >> > Tue May 27 22:47:00 UTC 2014 Setting up VPC virtual router system > >> > vm > >> > Tue May 27 22:47:00 UTC 2014 checking that eth0 has IP > >> > Tue May 27 22:47:00 UTC 2014 Setting up apache web server for VPC > >> > Tue May 27 22:47:00 UTC 2014 Enable service dnsmasq = 1 > >> > Tue May 27 22:47:00 UTC 2014 Enable service haproxy = 1 > >> > Tue May 27 22:47:00 UTC 2014 Processors = 1 Enable service = 0 > >> > Tue May 27 22:47:00 UTC 2014 Enable service cloud = 0 > >> > Tue May 27 22:47:00 UTC 2014 cloud: disable rp_filter > >> > Tue May 27 22:47:00 UTC 2014 disable rpfilter > >> > Tue May 27 22:47:00 UTC 2014 cloud: enable_fwding = 1 > >> > Tue May 27 22:47:00 UTC 2014 enable_fwding = 1 > >> > > >> > ifconfig (no eth1 shown) > >> > > >> > eth0 Link encap:Ethernet HWaddr 0e:00:a9:fe:03:5c > >> > inet addr:169.254.3.92 Bcast:169.254.255.255 > >> > Mask:255.255.0.0 > >> > > >> > eth2 Link encap:Ethernet HWaddr 02:00:7d:92:00:10 > >> > inet addr:10.0.1.1 Bcast:10.0.1.255 Mask:255.255.255.0 > >> > > >> > eth3 Link encap:Ethernet HWaddr 02:00:78:e9:00:05 > >> > inet addr:10.0.3.1 Bcast:10.0.3.255 Mask:255.255.255.0 > >> > > >> > lo Link encap:Local Loopback > >> > inet addr:127.0.0.1 Mask:255.0.0.0 > >> > > >> > > >> > cat /etc/network/interfaces > >> > auto lo eth0 > >> > iface lo inet loopback > >> > iface eth0 inet static > >> > address 169.254.3.92 > >> > netmask 255.255.0.0 > >> > > >> > lspci - shows 4 ehternet addapters > >> > ethtool eth1 = no link detected > >> > virsh - confirmed that eth1 is plugged to correct bridge > >> > (breth1-500) > >> > as > >> > indicated by management logs, and shows good MAC address as shown > >> > in > >> > managemetn log on pastebin.. > >> > > >> > This is completely makeing VPCs unusable... > >> > :( > >> > > >> > Cheers > >> > > >> > > >> > On 27 May 2014 16:36, Jayapal Reddy Uradi < > >> > jayapalreddy.ur...@citrix.com > >> > wrote: > >> > > >> > Hi, > >> > Can you please share management server and router logs in > >> > pastebin.comto understand the issue ? > >> > > >> > Thanks, > >> > Jayapal > >> > > >> > On 27-May-2014, at 6:21 PM, Andrija Panic < > >> > andrija.pa...@gmail.com> > >> > wrote: > >> > > >> > Hi, > >> > > >> > after the upgrade to ACS 4.3 (from 4.2.1) existing VRs for VPC > >> > lost > >> > their > >> > eth1 which is public NIC. VR got eth0(control nic) and eth2 and > >> > eth3 > >> > (bith > >> > belonging to Tiers). From CS GUI, it is reported that the VR has > >> > eth1 > >> > with > >> > Public network attached, but from inside (ssh to VR) there is no > >> > eth1 > >> > with > >> > public IP... > >> > > >> > Even after destroying those VR, they are recreated again, but > >> > without > >> > eth1. > >> > > >> > Anybody experienced same situtation ? > >> > > >> > Thanks, > >> > > >> > -- > >> > > >> > Andrija Panić > >> > -------------------------------------- > >> > > >> > > >> > > >> > > >> > -- > >> > > >> > > >> > > >> > > >> > > >> > > >> > -- > >> > Daan > >> > > >> > > >> > > >> > > >> > -- > >> > > >> > Andrija Panić > >> > -------------------------------------- > >> > http://admintweets.com > >> > -------------------------------------- > >> > > >> > > >> > > >> > -- > >> > Daan > >> > > >> > > >> > > >> > > >> > -- > >> > > >> > Andrija Panić > >> > -------------------------------------- > >> > http://admintweets.com > >> > -------------------------------------- > >> > > >> > >> > > > > > > -- > > > > Andrija Panić > > -------------------------------------- > > http://admintweets.com > > -------------------------------------- > > > > -- > Daan > -- Andrija Panić -------------------------------------- http://admintweets.com --------------------------------------